Revert commit to fix LUKS write errors

[tonyhutter]

Motivation and Context

Workaround ZFS on LUKS write errors on master branch.

Description

Revert bd7a02c as a workaround for #16631 and add a LUKS test case. This revert is more of a stop-gap for 2.3.x while we come up with a proper fix for #16631.

How Has This Been Tested?

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Performance enhancement (non-breaking change which improves efficiency)
• Code cleanup (non-breaking change which makes code smaller or more readable)
• Breaking change (fix or feature that would cause existing functionality to change)
• Library ABI change (libzfs, libzfs_core, libnvpair, libuutil and libzfsbootenv)
• Documentation (a change to man pages or other documentation)

Checklist:

• My code follows the OpenZFS code style requirements.
• I have updated the documentation accordingly.
• I have read the contributing document.
• I have added tests to cover my changes.
• I have run the ZFS Test Suite with this change applied.
• All commit messages are properly formatted and contain Signed-off-by.

[amotin]

Lets not hurry with this. If needed, we can use smaller workaround patch from #16631 (comment) .

[robn]

I'm not opposed to reverting (we haven't had this for most of 2.2 so it's a known quantity at least). I would prefer @amotin's far smaller patch if we think it covers enough (sounds right to me right now).

I will note that this will likely silence the uptick of recent issues, but it's not going to work around all related issues with LUKS, which have been seen in places that didn't have the the allocator change this patch reverts (late 2.2, and 2.1), because 4K blocks split across memory pages are still possible (gang ABDs, LINEAR_PAGE scatterlists). That's ok, reverting to a known and smaller broken is sensible, I just wanted anyone with an interest in those older problem to not get too excited by this change (based on the PR name).

[tonyhutter]

I'm putting this PR on hold in favor of the fix here:
#16631 (comment)

[robn]

Thanks @tonyhutter 👑

[mcmilk]

The luks sanity check is really fine.

[mcmilk]

This should be a tab.

[tonyhutter]

I'm closing this in favor of the workaround PR (#16678) and the standalone test case PR (#16681).