Dealing with single blockdev failure.

[star-39]

Hi team. I have a pool with 2 blockdevs.

$ stratis pool
Name                             Total Physical   Properties                                   UUID   Alerts
stratis-pool-1   8.19 TiB / 2.94 TiB / 5.25 TiB      ~Ca,~Cr   fc7cc699-82e6-46cc-8fa9-05f23283cd12

$ stratis blockdev
Pool Name        Device Node   Physical Size   Tier
stratis-pool-1   /dev/sda1          2.73 TiB   DATA
stratis-pool-1   /dev/sdd           5.46 TiB   DATA

$ stratis fs
Pool Name        Name      Size                          Created             Device                                UUID
stratis-pool-1   SPOOL_1   4 TiB / 2.93 TiB / 1.07 TiB   Feb 13 2022 16:18   /dev/stratis/stratis-pool-1/SPOOL_1   b74c306b-7377-45a8-894b-129d5a13357a

sdd was used to create the pool and initialized about 3TB of data, then added sda1.

No snapshot created ever.

Now, the problem is sda1 has failed.

$ dmesg
[ 9086.999366] device-mapper: thin: 253:18: unable to service pool target messages in READ_ONLY or FAIL mode
[ 9095.724451] device-mapper: thin: 253:18: unable to switch pool to write mode until repaired.

## dmesg is flooded by "device-mapper: thin: 253:18: unable to switch pool to write mode until repaired."

$ smartctl -a /dev/sda
SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed: read failure       90%     35414         502363895
# 2  Short offline       Completed: read failure       90%     35414         502363897

However, stratis reports nothing from this disk failure, no alerts issued.

The FS is inaccessible, ls command timeouts. If attempt to snapshot, following error was returned:

stratis fs snapshot stratis-pool-1 SPOOL_1 spool1_shot1
Execution failed:
stratisd failed to perform the operation that you requested. It returned the following information via the D-Bus: ERROR: failed to create spool1_shot1 snapshot for SPOOL_1 - DM Core error: low-level ioctl error due to nix error; header result: Some(DeviceInfo { version: Version { major: 4, minor: 45, patch: 0 }, data_size: 16384, data_start: 312, target_count: 0, open_count: 0, flags: (empty), event_nr: 0, dev: Device { major: 0, minor: 0 }, name: Some(DmNameBuf { inner: "stratis-1-private-fc7cc69982e646cc8fa905f23283cd12-thinpool-pool" }), uuid: None }), error: EOPNOTSUPP: Operation not supported on transport endpoint.

According to stratis-storage/stratisd#2570, current stratis data allocation strategy is similar to LVM, which data will be written to next disk(blockdev) after current disk is full. In this case, sdd is the "current" blockdev in pool.

It seems that stratis is not detecting the error from the failing spare disk (in this case sda1 which I guess it was never being utilized.)

Please test if this issue is reproducible.

If so, is there a option to grab the data out from the pool? The sdd one is healthy.

Thank you!

Some extra info:

OS: Fedora 35
$ stratis --version
3.0.1
$ stratis daemon version
3.0.4

[star-39]

No need to answer, stratis is too immature to use in production, no plan to deploy again.

The pool somehow became readable again after several reboots albeit dmesg is flooded with

[ 7380.718612] device-mapper: thin: 253:18: unable to switch pool to write mode until repaired.
[ 7390.731795] device-mapper: thin: 253:18: unable to switch pool to write mode until repaired.
[ 7400.742677] device-mapper: thin: 253:18: unable to switch pool to write mode until repaired.
[ 7410.750705] device-mapper: thin: 253:18: unable to switch pool to write mode until repaired.
[ 7420.761751] device-mapper: thin: 253:18: unable to switch pool to write mode until repaired.
[ 7430.770776] device-mapper: thin: 253:18: unable to switch pool to write mode until repaired.
[ 7440.780835] device-mapper: thin: 253:18: unable to switch pool to write mode until repaired.
[ 7450.793843] device-mapper: thin: 253:18: unable to switch pool to write mode until repaired.
[ 7460.805883] device-mapper: thin: 253:18: unable to switch pool to write mode until repaired.
[ 7470.815922] device-mapper: thin: 253:18: unable to switch pool to write mode until repaired.
[ 7480.823985] device-mapper: thin: 253:18: unable to switch pool to write mode until repaired.
[ 7490.834196] device-mapper: thin: 253:18: unable to switch pool to write mode until repaired.

Stratis is a step back from LVM2 in my opinion, very disappointed. It's 4 years from 1.0 release and nothing got improved. D-Bus does not matter at all to end user.

[bgurney-rh]

Hello,

If you are still able, would it be possible to show the metadata for the block devices used by the pool? This would be the first step toward diagnosing the pool's issue.

You can use the stratis-dumpmetadata program, which can be built from the stratisd repository, by checking out the release version (in your case, git checkout v3.0.4), and then running make stratis-dumpmetadata, followed by sudo ./target/debug/stratis_dumpmetadata /dev/sdd, and then again for /dev/sda1.

If you're not able to build the stratis-dumpmetadata program, there's an alternative way to show the block device metadata, via the hexdump program, to display the first 1048576 bytes of the block device, which contains the static header and MDA (metadata area):

$ sudo hexdump -C -n 1048576 /dev/sdd
$ sudo hexdump -C -n 1048576 /dev/sda1

For the output of each command, you should see output that looks similar to this:

00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000200  35 87 33 a6 21 53 74 72  61 30 74 69 73 86 ff 02  |5.3.!Stra0tis...|
00000210  5e 41 72 68 00 f8 ff 00  00 00 00 00 01 00 00 00  |^Arh............|
00000220  38 33 39 35 66 62 38 38  61 65 30 38 34 38 36 34  |8395fb88ae084864|
00000230  39 62 66 35 31 34 36 63  66 35 61 63 39 34 31 39  |9bf5146cf5ac9419|
00000240  63 36 66 37 64 36 65 61  37 39 31 65 34 61 31 37  |c6f7d6ea791e4a17|
00000250  61 37 33 65 34 61 63 35  65 34 30 33 63 62 37 64  |a73e4ac5e403cb7d|
00000260  f0 07 00 00 00 00 00 00  00 18 00 00 00 00 00 00  |................|
00000270  00 00 00 00 00 00 00 00  2e fd 31 62 00 00 00 00  |..........1b....|
00000280  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00001200  35 87 33 a6 21 53 74 72  61 30 74 69 73 86 ff 02  |5.3.!Stra0tis...|
00001210  5e 41 72 68 00 f8 ff 00  00 00 00 00 01 00 00 00  |^Arh............|
00001220  38 33 39 35 66 62 38 38  61 65 30 38 34 38 36 34  |8395fb88ae084864|
00001230  39 62 66 35 31 34 36 63  66 35 61 63 39 34 31 39  |9bf5146cf5ac9419|
00001240  63 36 66 37 64 36 65 61  37 39 31 65 34 61 31 37  |c6f7d6ea791e4a17|
00001250  61 37 33 65 34 61 63 35  65 34 30 33 63 62 37 64  |a73e4ac5e403cb7d|
00001260  f0 07 00 00 00 00 00 00  00 18 00 00 00 00 00 00  |................|
00001270  00 00 00 00 00 00 00 00  2e fd 31 62 00 00 00 00  |..........1b....|
00001280  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00002000  96 d2 5b ef 65 73 5e 87  f1 01 00 00 00 00 00 00  |..[.es^.........|
00002010  2f fd 31 62 00 00 00 00  b6 ed a2 06 01 01 00 00  |/.1b............|
00002020  7b 22 6e 61 6d 65 22 3a  22 73 70 6f 6f 6c 31 22  |{"name":"spool1"|
00002030  2c 22 62 61 63 6b 73 74  6f 72 65 22 3a 7b 22 64  |,"backstore":{"d|
00002040  61 74 61 5f 74 69 65 72  22 3a 7b 22 62 6c 6f 63  |ata_tier":{"bloc|
00002050  6b 64 65 76 22 3a 7b 22  61 6c 6c 6f 63 73 22 3a  |kdev":{"allocs":|
00002060  5b 5b 7b 22 70 61 72 65  6e 74 22 3a 22 63 36 66  |[[{"parent":"c6f|
00002070  37 64 36 65 61 2d 37 39  31 65 2d 34 61 31 37 2d  |7d6ea-791e-4a17-|
00002080  61 37 33 65 2d 34 61 63  35 65 34 30 33 63 62 37  |a73e-4ac5e403cb7|
00002090  64 22 2c 22 73 74 61 72  74 22 3a 38 31 39 32 2c  |d","start":8192,|
000020a0  22 6c 65 6e 67 74 68 22  3a 31 36 37 36 35 35 32  |"length":1676552|
000020b0  30 7d 5d 5d 2c 22 64 65  76 73 22 3a 5b 7b 22 75  |0}]],"devs":[{"u|
...

[elrubaqur]

I am curious about the expected behavior in this case. Once a blockdev is added to the pool, is the expectation that the blockdev must stay available in order for the pool to be useable?

I've read the following related discussions to this topic:
stratis-storage/stratisd#2570 and #503

Reviewing the OP's configuration:

Name                             Total Physical   Properties                                   UUID   Alerts
stratis-pool-1   8.19 TiB / 2.94 TiB / 5.25 TiB      ~Ca,~Cr   fc7cc699-82e6-46cc-8fa9-05f23283cd12

$ stratis blockdev
Pool Name        Device Node   Physical Size   Tier
stratis-pool-1   /dev/sda1          2.73 TiB   DATA
stratis-pool-1   /dev/sdd           5.46 TiB   DATA

$ stratis fs
Pool Name        Name      Size                          Created             Device                                UUID
stratis-pool-1   SPOOL_1   4 TiB / 2.93 TiB / 1.07 TiB   Feb 13 2022 16:18   /dev/stratis/stratis-pool-1/SPOOL_1   b74c306b-7377-45a8-894b-129d5a13357a

Since the data is written sequentially and not-striped, my initial assumption is that the data on the first disk should still be available since the files were fully written to a single disk. In the OP's case, /dev/sdd contained all the data, so even if /dev/sda1 had failed, given that "data will be written to the first device and when that device is full, data will be written to the next device", then there should be some way to theoretically recover all of that data, is that correct?

For example, would he have been able to mount the underlying XFS partition directly in read-only mode and copy out his data? This assumes that the "first device" as written in stratis-storage/stratisd#2570 means the first device added into the pool in chronological order, and not some other method of determine "first" whereby /dev/sda1 can some how become first, and then new data was then written to /dev/sda1.

And if his /dev/sdd was in fact full and new data had to be written to /dev/sda1, then perhaps the only loss of data would have been the last file written to /dev/sdd which did not fully fit on /dev/sdd and so part of the file existed on /dev/sda1?

Essentially what I'm getting at here is this question: If I were to create a stratis pool with N number of blockdevs, where each blockdev is a physical hard disk, will 1 disk failure render the entire pool unusable? I understand much of these are "what if?" scenarios, but I am interested in learning the failure cases to best configure a stratis pool for my use case and my acceptable trade offs. Thanks!

[jbaublitz]

The high level is that we are working on software RAID now but it is not yet released. Until that is released, it is recommended to use hardware or software RAID array underneath the Stratis pool if you want to have resilience against disk failure. Theoretically you may be able to recover partial data from the devices that haven't failed, but we make no guarantees that you will be able to, so our recommendation is to use a data protection layer like RAID if you want to be able to recover from disk failures.