Plotting using solely HDDs

[j4r3kb]

I have 4x Velociraptor 2.5'' 10k 1TB drives and was plotting on them with Swar Manager in the standard process in parallel 1 plot per drive and they were finishing the plot in 6.5 hours.

I tried this plotter just for fun (wasn't expecting miracles on HDD) and here are the results for single run with 1 plot, for temp, temp2 and final separate HDD was used. 8 threads.

./build/chia_plot -r 8 -t /media/raptor2/ -2 /media/raptor3/ -d /media/raptor4/ -p xxx -f xxx
Multi-threaded pipelined Chia k32 plotter - 18f4918
Final Directory: /media/raptor4/
Number of Plots: 1
Process ID: 11685
Number of Threads: 8
Number of Buckets: 2^7 (128)
Pool Public Key:   xxx
Farmer Public Key: xxx
Working Directory:   /media/raptor2/
Working Directory 2: /media/raptor3/
Plot Name: plot-k32-2021-06-10-22-16-7397106a1462efb12655900c8d217bcc35cdc72371640fdcd3f494487ff37aa2
[P1] Table 1 took 183.253 sec
[P1] Table 2 took 714.764 sec, found 4295106355 matches
[P1] Table 3 took 1173.22 sec, found 4295092961 matches
[P1] Table 4 took 1385.6 sec, found 4295174532 matches
[P1] Table 5 took 1200.97 sec, found 4295168983 matches
[P1] Table 6 took 1022.64 sec, found 4295131175 matches
[P1] Table 7 took 710.215 sec, found 4295155595 matches
Phase 1 took 6391.04 sec
[P2] max_table_size = 4295174532
[P2] Table 7 scan took 234.217 sec
[P2] Table 7 rewrite took 518.844 sec, dropped 0 entries (0 %)
[P2] Table 6 scan took 127.722 sec
[P2] Table 6 rewrite took 363.486 sec, dropped 581390214 entries (13.536 %)
[P2] Table 5 scan took 137.85 sec
[P2] Table 5 rewrite took 353.2 sec, dropped 762127984 entries (17.7438 %)
[P2] Table 4 scan took 140.129 sec
[P2] Table 4 rewrite took 343.917 sec, dropped 829063661 entries (19.3022 %)
[P2] Table 3 scan took 145.449 sec
[P2] Table 3 rewrite took 353.182 sec, dropped 855175847 entries (19.9105 %)
[P2] Table 2 scan took 143.432 sec
[P2] Table 2 rewrite took 350.164 sec, dropped 865648238 entries (20.1543 %)
Phase 2 took 3219.94 sec
Wrote plot header with 268 bytes
[P3-1] Table 2 took 286.84 sec, wrote 3429458117 right entries
[P3-2] Table 2 took 367.799 sec, wrote 3429458117 left entries, 3429458117 final
[P3-1] Table 3 took 349.603 sec, wrote 3439917114 right entries
[P3-2] Table 3 took 365.751 sec, wrote 3439917114 left entries, 3439917114 final
[P3-1] Table 4 took 353.39 sec, wrote 3466110871 right entries
[P3-2] Table 4 took 367.907 sec, wrote 3466110871 left entries, 3466110871 final
[P3-1] Table 5 took 382.912 sec, wrote 3533040999 right entries
[P3-2] Table 5 took 373.093 sec, wrote 3533040999 left entries, 3533040999 final
[P3-1] Table 6 took 396.455 sec, wrote 3713740961 right entries
[P3-2] Table 6 took 422.121 sec, wrote 3713740961 left entries, 3713740961 final
[P3-1] Table 7 took 740.048 sec, wrote 4295155595 right entries
[P3-2] Table 7 took 504.198 sec, wrote 4294967296 left entries, 4294967296 final
Phase 3 took 4911.38 sec, wrote 21877235358 entries to final plot
[P4] Starting to write C1 and C3 tables
[P4] Finished writing C1 and C3 tables
[P4] Writing C2 table
[P4] Finished writing C2 table
Phase 4 took 118.035 sec, final plot size is 108835884324 bytes
Total plot creation time was 14640.4 sec
Started copy to /media/raptor4/plot-k32-2021-06-10-22-16-7397106a1462efb12655900c8d217bcc35cdc72371640fdcd3f494487ff37aa2.plot
Copy to /media/raptor4/plot-k32-2021-06-10-22-16-7397106a1462efb12655900c8d217bcc35cdc72371640fdcd3f494487ff37aa2.plot finished, took 677.453 sec, 153.212 MB/s avg.

So 4 hours. Not worth it for my case to change the plotting process.

Now, if someone has any suggestions for another test I can do it. Maybe with 4 HDDs I could try using the same drive for temp and temp2 assign 2 threads and run 4 of the same tasks with the same destination HDD. So 8 threads total writing to common destination. What do you guys think?

[dyn4mic]

I had success with 8x146 15k SAS drives and raid0 them together, still figuring out the relationship between tmp1 and tmp2.
at the moment 3 disks tmp1 5 disks tmp2 gives me <5000s plot time. --> ~17,28 plots/day
beforehand i tried the relation ship of tmp1 to tmp2 2:6 disks gave me around ~5150s plot time --> 16,77 plots/day
until then i used Swar with 4 parallel plots mdadm 2 drives each ~7.5h per plot --> 12plots/day
currently playing around on just bundling all drives together and using just one tmp dir.
i am on a e5 2678 v3(12c/24t) with ddr3 32gb ram

[j4r3kb]

Today I have run 4 parallel plotter processes 1 plot per HDD drive, temp1 and temp2 being the same, all to common destination directory.
And finish times are 5.5 - 6 hours (including copying to destination) so up to 1 hour faster than standard process.
Now I will explore buckets less than 128.

[dyn4mic]

So my conclusion, although waste of space, in my setup a striped zfs pool of my 8 disks brings the best performance, around 4700s per plot.
Concepts are different, madmax plotter relies on fast Io and tries to saturate it, if you don't have that, same as I, and Io stays bottleneck the benefit isn't that big.
I will play around and will still compare parallel plots as well.

[j4r3kb]

sudo lvcreate --type raid0 --stripes 8 --stripesize 1024 -l 100%free -y -n chiatmp md0

is this line correct? around -l 100%free -y?

[dyn4mic]

yes -y parameter to skip a prompt of accepting overwrite of other fs signature

[j4r3kb]

I mean 100%free strange value ;)

[dyn4mic]

yeah that one as well, allocating 100% of free space full in line with spec https://linux.die.net/man/8/lvcreate

[j4r3kb]

Using your tips on xfs and raid0 I have created volume from 3 x WD1000DHTZ and 1 x ST2000NC001.
Run 1 plot with 6 threads, 128 buckets.

Multi-threaded pipelined Chia k32 plotter - f495816
Final Directory: /media/farm/plot/
Number of Plots: 1
Crafting plot 1 out of 1
Process ID: 8942
Number of Threads: 6
Number of Buckets: 2^7 (128)
Pool Public Key:   xxx
Farmer Public Key: xxx
Working Directory:   /media/chiatmp/
Working Directory 2: /media/chiatmp/
Plot Name: plot-k32-2021-06-16-23-54-345c5d3cb99ae9f5cb7d377d4f6efe0b018f415069f3ba6f8249fb6fb0bfa7df
[P1] Table 1 took 59.5048 sec
[P1] Table 2 took 300.973 sec, found 4294928466 matches
[P1] Table 3 took 458.932 sec, found 4294971347 matches
[P1] Table 4 took 546.559 sec, found 4294848206 matches
[P1] Table 5 took 494.747 sec, found 4294667963 matches
[P1] Table 6 took 429.475 sec, found 4294342772 matches
[P1] Table 7 took 290.852 sec, found 4293791925 matches
Phase 1 took 2581.87 sec
[P2] max_table_size = 4294971347
[P2] Table 7 scan took 80.2514 sec
[P2] Table 7 rewrite took 168.154 sec, dropped 0 entries (0 %)
[P2] Table 6 scan took 54.9719 sec
[P2] Table 6 rewrite took 148.112 sec, dropped 581335561 entries (13.5372 %)
[P2] Table 5 scan took 49.2346 sec
[P2] Table 5 rewrite took 133.652 sec, dropped 762012975 entries (17.7432 %)
[P2] Table 4 scan took 64.1322 sec
[P2] Table 4 rewrite took 132.591 sec, dropped 828886357 entries (19.2995 %)
[P2] Table 3 scan took 48.3343 sec
[P2] Table 3 rewrite took 139.585 sec, dropped 855099123 entries (19.9093 %)
[P2] Table 2 scan took 47.7997 sec
[P2] Table 2 rewrite took 142.889 sec, dropped 865532370 entries (20.1524 %)
Phase 2 took 1214.42 sec
Wrote plot header with 268 bytes
[P3-1] Table 2 took 176.591 sec, wrote 3429396096 right entries
[P3-2] Table 2 took 159.621 sec, wrote 3429396096 left entries, 3429396096 final
[P3-1] Table 3 took 205.689 sec, wrote 3439872224 right entries
[P3-2] Table 3 took 162.211 sec, wrote 3439872224 left entries, 3439872224 final
[P3-1] Table 4 took 201.106 sec, wrote 3465961849 right entries
[P3-2] Table 4 took 159.102 sec, wrote 3465961849 left entries, 3465961849 final
[P3-1] Table 5 took 204.055 sec, wrote 3532654988 right entries
[P3-2] Table 5 took 152.147 sec, wrote 3532654988 left entries, 3532654988 final
[P3-1] Table 6 took 216.602 sec, wrote 3713007211 right entries
[P3-2] Table 6 took 168.003 sec, wrote 3713007211 left entries, 3713007211 final
[P3-1] Table 7 took 250.907 sec, wrote 4293791925 right entries
[P3-2] Table 7 took 220.352 sec, wrote 4293791925 left entries, 4293791925 final
Phase 3 took 2282.69 sec, wrote 21874684293 entries to final plot
[P4] Starting to write C1 and C3 tables
[P4] Finished writing C1 and C3 tables
[P4] Writing C2 table
[P4] Finished writing C2 table
Phase 4 took 91.4301 sec, final plot size is 108820310880 bytes
Total plot creation time was 6170.45 sec (102.841 min)
Started copy to /media/farm/plot/plot-k32-2021-06-16-23-54-345c5d3cb99ae9f5cb7d377d4f6efe0b018f415069f3ba6f8249fb6fb0bfa7df.plot
Copy to /media/farm/plot/plot-k32-2021-06-16-23-54-345c5d3cb99ae9f5cb7d377d4f6efe0b018f415069f3ba6f8249fb6fb0bfa7df.plot finished, took 321.711 sec, 322.585 MB/s avg.

So around 14 plots per day, running parallel with standard method 1 per drive results in more plots.

[gryan315]

I ran a test with 16x 10k RPM drives in a raid0 array, saw around 2k r/w speeds while it was plotting. Array made with mdadm, formatted in xfs. The drives are 10x 600gb 64mb cache, and 6x 900gb 128mb cache, Just used a 100G partition from each in the array. 2x xeon 2678v3.

48 threads
128 buckets
Phase 1: 717.045 sec
Phase 2: 352.675 sec
Phase 3: 518.501 sec
Phase 4: 100.896 sec
Total: 1689.21 sec

For reference, using RAM tmpfs with 48 threads I only got 1566.97 sec total time.

[madMAx43v3r]

So as you can see, phase 1 and 4 are unaffected, phase 2 is faster with logical cores, and phase 3 is faster with physical cores. Phase 3's speed up is more than phase 2's slow down, so slight improvement overall with physical cores as the -r value. I guess optimal would be if you could set phase 2 to use double the multiplier of the rest of the job.

See #611

[gryan315]

Sounds exciting

[cojarbi]

I see - k was added, how what are the parameter options for it?

[gryan315]

I see - k was added, how what are the parameter options for it?

I haven't had a chance to test it yet, but it would appear to me that you want to set it to 2 to double phase 2 -r value.

[gryan315]

Just finished the first plot using -K, it works as expected. Using -r 24 and -k 2 resulted in phase 2 times the same as -r 48, while maintaining the faster phase 3 time with -r 24. New fastest plot for my rig.

[cojarbi]

So i got my new plotter a Dell R Dual Xeon E5-2670 v2 with 128GB ram and SAS 15k HDD. Using tmpfs and hardware raid0 on a h710 mini.

My first set of testing brings speed improvements over my previous one with this results:

256 -r 40
787/546/639/71 > 2043

256+512 -r 40
796/563/700/58 > 2117

512 -r 40
823/644/715/57> 2239

512+256 -r 40
846/590/691/68> 2195

Now i started testing -r 20 and its the same average result

[cojarbi]

I se a difference of about 2 min less when using 36 and about the same as before when using 30. I need to try clearing my hardware raid and using mdadm

[gryan315]

Can you do that with an h710, thought it was raid only? Are you planning to make a bunch of single drive raid 0 arrays on the h710 and then mdadm them in linux? If possible, I'd recommend swapping out the h710 for an h310 mini, or if you have a full size pcie an h200, which is what I use (the h200e external version). I'm not sure if there is a significant performance difference between hardware raid and software raid on a HBA, at least not when using the older hardware it was designed to be used with.

[cojarbi]

i have another server with a h310, not much difference. Yes you can flash most of those cards, this works and tested and its reversible, simple boot and flash https://fohdeesha.com/docs/perc/

The reason i say this is that i was able to see max w/r speed with mdadm in the 2000 range and with the cards is at 600 maybe

[cojarbi]

I haven't been able to test much, the server freezes when im sudo mkfs.xfs -m crc=0 /dev/md11

[gryan315]

You can try without the -m crc=0, in my quick testing I saw no significant advantage to using it.