Command Reference for config Write-back vs Write-through cache

[timwhite1]

I can see verbiage of support for both modes but I have not located any man page verbiage or articles on how to do this.

From existing docs:
"Caching may be configured for write-back and write-through modes."

Is there is simple command reference that can be shared?

[timwhite1]

I've reviewed a large amount of the stratisd and stratis-cli code and so far am not seeing any references to write-back or write-through modes. So far all reviewed caching segments and references seem to only pass through enablement but are not looking for write mode arguments.

Based on the early docs notating this functionality, it appears this is a planned implementation to gain interest and forecast eventual feature set, but is not there yet.

If write caching is enabled via the typical LVM cache setup my assumption would be it is defaulting to write-through cache. But it is unclear if write cache is being enabled at all.

[mulkieran]

Sorry we didn't get back to you yesterday; your conclusions all look correct. We do not currently use dm-writecache, although we have been aware of it. The current implementation of the Stratis cache is alway write-through, the default. We'll be fixing up the design doc shortly to reflect better the true state of affairs.

[timwhite1]

That's helpful - thank you for the quick response!

So to clarify, currently Stratis Cache is read and write cache in similar form as dm-cache, and is write-through only. And write-back is not a configurable option yet?

[mulkieran]

Exactly.

[timwhite1]

Thank you!

[timwhite1]

Would there be a way to apply write-back cache in Stratis for testing purposes via an unsupported cli or other method? I am exploring multiple storage options but seem to keep returning to Stratis and traditional LVM. I would love to help test this setup if possible.

[drckeefe]

@timwhite1
Here is an example of changing the write mode for testing, though it is not supported.
I hope the output explains the process.

# stratis pool create p1 /dev/nvme0n1p1
# stratis cache init p1 /dev/nvme0n1p2
# stratis pool init-cache p1 /dev/nvme0n1p2
# dmsetup status
stratis-1-private-4b4114c79399430980581093ca2d9175-flex-mdv: 0 1048576 linear
stratis-1-private-4b4114c79399430980581093ca2d9175-flex-thindata: 0 18466816 linear
stratis-1-private-4b4114c79399430980581093ca2d9175-flex-thinmeta: 0 1376 linear
stratis-1-private-4b4114c79399430980581093ca2d9175-physical-cache: 0 19518144 cache 8 30/131072 2048 1/8067 0 0 5 2 0 1 0 1 writethrough 2 migration_threshold 2048 smq 0 rw -
stratis-1-private-4b4114c79399430980581093ca2d9175-physical-cachesub: 0 16521216 linear
stratis-1-private-4b4114c79399430980581093ca2d9175-physical-metasub: 0 1048576 linear
stratis-1-private-4b4114c79399430980581093ca2d9175-physical-originsub: 0 19518144 linear
stratis-1-private-4b4114c79399430980581093ca2d9175-thinpool-pool: 0 18466816 thin-pool 0 26/172 0/9017 - rw no_discard_passdown error_if_no_space - 43
# dmsetup table stratis-1-private-4b4114c79399430980581093ca2d9175-physical-cache
0 19518144 cache 253:8 253:9 253:3 2048 1 writethrough default 0
# dmsetup reload stratis-1-private-4b4114c79399430980581093ca2d9175-physical-cache --table '0 19518144 cache 253:8 253:9 253:3 2048 1 writeback default 0'
# dmsetup suspend stratis-1-private-4b4114c79399430980581093ca2d9175-physical-cache
# dmsetup resume stratis-1-private-4b4114c79399430980581093ca2d9175-physical-cache
# dmsetup status
stratis-1-private-4b4114c79399430980581093ca2d9175-flex-mdv: 0 1048576 linear
stratis-1-private-4b4114c79399430980581093ca2d9175-flex-thindata: 0 18466816 linear
stratis-1-private-4b4114c79399430980581093ca2d9175-flex-thinmeta: 0 1376 linear
stratis-1-private-4b4114c79399430980581093ca2d9175-physical-cache: 0 19518144 cache 8 40/131072 2048 1/8067 0 0 41 2 0 0 1 1 writeback 2 migration_threshold 2048 smq 0 rw -
stratis-1-private-4b4114c79399430980581093ca2d9175-physical-cachesub: 0 16521216 linear
stratis-1-private-4b4114c79399430980581093ca2d9175-physical-metasub: 0 1048576 linear
stratis-1-private-4b4114c79399430980581093ca2d9175-physical-originsub: 0 19518144 linear
stratis-1-private-4b4114c79399430980581093ca2d9175-thinpool-pool: 0 18466816 thin-pool 0 26/172 0/9017 - rw no_discard_passdown error_if_no_space - 43

[timwhite1]

Excellent - thank you! I will test this setup as well. I appreciate the quick response!

[timwhite1]

I had some time to test this and wanted to share a little unexpected feedback. I did not have prior LVM / DM-Cache writeback experience and really had the wrong expectations. A few takeaways for others who want to test write Caching on Stratis:

1. In Write-back mode, data has to be deemed "hot" before it will write directly to Cache volumes

   • this is inherited behavior from the underlying DM-Cache and will be exhibited in LVM Cache as well

   • from the standpoint of SSD drive endurance this is good

   • but combined w/ LVM Integrity RAID HDD origin disks, write performance is still poor

     • in my testing, initial writes stayed around 45-60 MB/s
       (on 2 x new Seagate Exos SATA 7k 10TB drives in LVM Integrity Mirror)

     • repeat writes of the same data get cached and begin to hit higher speeds
       (140 - 260 MB/s and eventually 400-700 on repeat trials)

     • read speeds are amazing the whole time leveraging cache for much of recent data
       (1-13 GB/s on 2 x new WD SN850 NVMe drives in LVM Integrity Mirror)

     • These are quick stats from Gnome Disks Utility Benchmarks

     • FIO could add more complete IO stats but this was enough to assess general performance

     • based on my spindle count and use case this is not enough of a write performance boost
       to make the workload usable on Integrity RAID so I will need to keep looking

2. After a reboot any manual DM-Cache table writeback config will revert to native Stratis writethrough

   • so even if hot write caching is enough you'll have to reconfig post reboot every time
   • this manual override is for temporary testing only and is not supported for reasons like this
   • at some point Stratis will likely support all of the above natively (Integrity RAID, Writeback, Writecache, etc).

3. The LVM writecache option may be a better alternative for always caching writes

   • a small writecache LV could be applied to the 10TB mirror origin volume
     • it seems unlikely LVM rules will allow Integrity RAID with Writecache but maybe via direct DM-Cache config...
   • then when imported into the Stratis Pool the slow integrity mirror writes could be overcome
   • with Stratis read caching applied you could get back to optimal performance most of the time
   • based on previous benchmarks should be in the neighborhood of:
     • 600-700 MB/s writes (hot or cold)
     • 1-13 GB/s hot reads
     • 200-400 MB/s cold reads
   • this would be completely usable for my workload

[jbaublitz]

Just to make sure I'm understanding, are saying dm-writecache support is better for your workflow than writeback in dm-cache? If so, this is consistent with what I've seen discussed about dm-cache targeting read-heavy workflows and dm-writecache being more optimized for write-heavy workflows. I'll double check with the team, but we've discussed this as a future feature, so we likely do eventually want to support this.

[timwhite1]

My primary need for dm-writecache is to overcome the write performance hit of dm-integrity on the origin volume. But I wouldn't say this is actually a write heavy work load.

dm-writecache should be able to overcome this, where write-back is not enough until data begins to warm. That means all initial writes are at the low speed of the origin disks which was unexpected, and just too low on an Integrity RAID origin volume.

If there were a future way to get both together in Stratis that would really be helpful.

Where I would normally see 260-300 MB/s write speeds on a standard RAID 1 (2 x 7k), it drops to 45-60 MB/s when configured as Integrity RAID 1.

NVMEe Integrity RAID 1 is equally impacted but it is felt less due to the higher performing medium. For example standard RAID 1 (2 x WD SN850) write speeds are around 6.5 - 7.5 GB/s, but drop to 600-700 MB/s in an Integrity RAID 1. With that kind of performance I'm still fine using Integrity RAID on my Cache volume but some might not be.

The ideal solution would be to improve write performance of dm-integrity which I know is out of scope here but is actually the root cause. Or to include a new operator in dm-cache writeback mode to always send newly written data to a given portion of the cache volume. For example 90 / 10 could be great -- use 90% for standard hot read / writes and reserve 10% for cold writes which will be eventually flushed to origin volume. When the origin volume is overly slow on writes this would ensure high performance basically all the time.

Based on current Stratis and LVM feature set I plan to layer something similar to the following to get where I need on performance in the short term:

• Stratis with basic Write-through Cache
• LVM Writecache
• LVM Integrity RAID (or DM-Integrity + RAID)

I didn't expect to see a warm-up required on Write-back cache. But if I can combine Stratis caching with an underlying LVM Writecache volume that will likely get me where I want on performance.

I know BTRFS or ZFS could solve the Integrity / checksum performance impact more readily and likely negate the need for dedicated writecache. But I want to ensure RAID uptime even when a drive fails, and BTRFS can't really provide that (manual remediation required to mount in degraded state). And I don't want to be limited to a short list of officially supported kernel builds / platforms with direct ZFS support, or have to carefully plan my system patching to avoid kernel / ZFS module incompatibility. Native tools have a big appeal and it's great what Stratis is doing to simplify and unify to make these features more accessible.

I can see a future where much of this will be configurable graphically via Cockpit or other UI and grow to be used by admins and enthusiasts of all levels. Based on device mapper and LVM foundations there is already a history of stability so it's not like anyone is taking a big leap of faith on an experimental storage system. And even if it takes a while to get there, slow change is often lasting change.

[timwhite1]

Little update here...

I initially tried to design with this layout, using LVM writecache above the integrity RAID and below the Stratis pool:

• Unfortunately this collides with current LVM rules blocking Integrity + Caching combinations

I'm going to attempt this new design with dm-writecache devices above each backing volume and below the LVM Integrity RAID:

• Really hoping this passes the rules check as the next iteration would likely be dm-writecache + dm-integrity + mdadm RAID (plus Stratis) which seems pretty over the top complex.

Big picture, this helps to illustrate the challenges that Stratis is beginning to solve-- simplifying complex layered configs into one interface.

[drckeefe]

Thank you @timwhite1 for spending the time on this. It might take us a couple of weeks to review this, as we are near the end of a testing and release phase. This is interesting work you are doing, so please allow us time to respond.

[timwhite1]

No pressure! And thanks for the reply. I'm going a little different route than the above visuals and will post an update after more testing.

[timwhite1]

@drckeefe I wanted to check in and see if there are any testing builds I can experiment on with DM-writecache mode enabled in Stratis? (Caching all writes as compared to writeback or write-through)

I noticed encrypted read Caching was released recently which is a great breakthrough as well. I will say again, the simplicity that Stratis is working toward is so compelling.

[drckeefe]

@jbaublitz @mulkieran thoughts on adding writecache? I personally would like to see this feature added in. My guess is that it will have to come after metadata rewrite, RAID, re-encrypt, and integrity (might be some other minor things thrown in there too). I believe this isn't a minor feature addition because dm-writecache is a different device mapper target from dm-cache and requires backend plumbing. I would also like to see a feature to remove a cache if needed. Thanks for keeping up on this feature.