Disable Copy-on-Write for filesystem

[timwhite1]

Is there a supported or testing method to disable copy-on-write for select filesystems within a pool?

I would like to use Stratis to manage all of my storage, while segmenting filesystems for a given use case. For example:

• filesystem1 = general SMB storage (COW)
• filesystem2 = DB (non COW)
• filesystem3 = VM disk storage (non COW)

Copy-on-write is great for some workloads and not for others. If there is a different way to implement such as the folder or file level I can work with that as well.

Thank you for your help!

[mulkieran]

Could you explain how you feel the XFS CoW functionality affects you? Stratis generally accepts XFS defaults when making files; when reflink capability became the default for XFS then it became the default for Stratis. But it seems that there isn't much cost to having reflink enabled, just some additional metadata on filesystem creation which ought to be negligible in the context of the other filesystem metadata, which is itself pretty negligible compared to the filesystem data. And the user can control whether or not a copy is CoW with the --reflink option, so simply enabling reflink for the filesystem does not require using it.

[jbaublitz]

Just to clarify, in this statement below it appears I can manually not use reflinks by using the right operator in the cp command. But otherwise all copy or modification operations to a file, DB, VM image, etc. will default to reflink (cow) functionality. Am I understanding correctly? (If you have a link that explains that is great too)
"By default, we enable reflink support in XFS by accepting the defaults, but it is up to you to actually use it through the cp command or an equivalent ioctl (there's a reflink flag in cp that will reflink the target and destination extents which will trigger CoW if either diverges)."

It's actually the exact opposite. Let me try to give you examples to better explain this.

cp one_file.txt another_file.txt does not reflink and so when either is changed, there is no copy on write. They are both already two entirely separate files that were fully copied to two independent locations on the filesystem when you invoked the command.

cp --reflink=always one_file.txt another_file.txt reflinks the file and because of this, the full file is not copied when this command is run. However, if either of the two reflinked copies diverges from the other (there is a write operation to either), then CoW will be triggered.

So the short version is:

• CoW is opt-in only
• You opt in on a per file basis by using cp --reflink=always
• Our metadata accepts XFS defaults which enables reflink (but reflink is not the default) and allows you to opt in at any time for any file
• Any regular copy operation without the --reflink flag in cp or equivalent ioctl() will not be CoW

Even shorter version is:

• Any file that is not created via cp --reflink=always will not be CoW

Does that clear it up a little?

[timwhite1]

That's helpful thank you! I think COW performance should not be an issue at all and should not be anything we need to try to work around in Stratis.

[sandeen]

The above is not correct;--reflink=auto has been the default in cp for the past couple years, after this commit to coreutils:

coreutils/coreutils@25725f9

To see this in action, on reflink-capable XFS:

$ rpm -qf `which cp`
coreutils-8.32-36.fc35.x86_64
$ echo foo > file_a
$ cp file_a file_b
$ xfs_bmap file_a file_b
file_a:
	0: [0..7]: 262662080..262662087
file_b:
	0: [0..7]: 262662080..262662087

As you can see, they are sharing blocks after the cp.

[jbaublitz]

@sandeen Thanks for the correction! I must have been looking at outdated documentation.

@timwhite1 You'd want cp --reflink=never to explicitly tell cp not to reflink.

[timwhite1]

That's helpful. Thank you both!

I really don't see copy based reflinks posing performance issues either. My only concern was if any existing file (DB or VM disk image) when changed, would create endless new version pointers for each write. But it does not sound like that is the case. I want that periodically via snapshots but not continuously as some other systems do.

On the other hand when I want to clone a large file, that is really helpful that XFS and Stratis default to reflink to save space. Very cool. Thanks again!

[sandeen]

My guess is that this may stem from confusion about how btrfs operates (overwriting any file block will COW those blocks) vs how XFS operates (overwriting an extent shared between reflinked files will COW those blocks.)

btrfs has a "nocow mode" specifically to avoid that behavior for workloads like databases; this is not something that is needed for XFS. If you have not reflinked your database files on XFS, there is no COW on overwrite. (And if you have reflinked them, then COW is the only possible behavior, which is true even on btrfs in "nocow" mode; it is the only way to disassociate a shared extent when one file changes.)

[timwhite1]

This is helpful. Yes my Qs STEM from recommendations on BTRFS not to use COW for DB or VM workloads. And even on ZFS there can be a negative performance impact without adequate hardware added to compensate for the COW impact.

If you all are saying that Stratis can offer COW and low impact snapshots, with high performance on all kinds of workloads (including VMs and DBs) this is game changing.

Once we get an automated snapshot integration like snapper or something similar, there really will be little motivation to deploy BTRFS or ZFS.

I've been testing with LVM Integrity raid for the base LV imported into the Stratis pool. And I'm getting ready to test with an NVMe integrity mirror for caching. I've been doing a single drive cache volume but want redundancy for write-back to be a little safer.

If all goes well this will become a high performance, resilient combo I can deploy in basically any environment, regardless of distro. Things are really looking up in the world of Linux storage!