Multiple pool support for ztest

[don-brady]

Motivation and Context

Many zfs deployments are using multiple pools. However, ztest only tests with a single pool. It would be nice to be able to run ztest against multiple pools at the same time to help expose additional issues when multiple pools are in play.

Description

Adds support, ztest --pool-count <poolcount> for testing ztest with multiple pools.
Note that the --threads count (default 23) in ztest is per pool.

Example

$ ztest –-pool-count 7 -K raidz -m 0 -r 6 -VVVV
7 pools, 5 vdevs, 7 datasets, 23 threads, 6 raidz disks, parity 1, 300 seconds...

...

Pass   3,  SIGKILL,  63% done,    1m50s to go
  ztest-pool-0:   0 ENOSPC,  1.4% of 4.12G used, 18 leaf vdevs
  ztest-pool-1:   0 ENOSPC,  3.9% of 1.38G used, 6 leaf vdevs
  ztest-pool-2:   0 ENOSPC,  5.4% of 1.38G used, 6 leaf vdevs
  ztest-pool-3:   0 ENOSPC,  3.3% of 1.62G used, 12 leaf vdevs
  ztest-pool-4:   0 ENOSPC,  4.6% of 1.38G used, 6 leaf vdevs
  ztest-pool-5:   0 ENOSPC,  7.5% of 1.62G used, 6 leaf vdevs
  ztest-pool-6:   0 ENOSPC,  6.0% of 1.38G used, 12 leaf vdevs

Sponsored-by: Klara, Inc.
Sponsored-by: Wasabi Technology, Inc.

How Has This Been Tested?

Ongoing testing with/without multiple pools and using zloop

Notes for reviewers

1. Changed DEFAULT_VDEV_DIR from "/tmp" to "/tmp/ztest" since with multiple pools, the vdev file count can be large.

2. Abstracted the generation of disk path strings under a make_vdev_path().

3. Added ztest_check_disk_space(). Since vdev files are initially sparse, make sure there's space for at least half of the vdev size for all the vdevs we instantiate when we start and when we create new vdevs. With multiple pools the leaf vdev file count can grow fast and consume lots of space.

4. Added per-pool runtime state for the child process (does not persist when child process is killed).

typedef struct ztest_pool {
        char            *zp_name;
        spa_t           *zp_spa;
        ztest_ds_t      *zp_ds;
        kthread_t       *zp_resume_thread;
        kthread_t       *zp_deadman_thread;
        kmutex_t        zp_vdev_lock;
        kmutex_t        zp_checkpoint_lock;
        boolean_t       zp_device_removal_active;
        boolean_t       zp_pool_scrubbed;
        ztest_cb_list_t zp_commit_cb_list;
        uint64_t        zp_commit_min_txg_delay;
        int             zp_commit_cb_counter;
} ztest_pool_t;

5. Added a new shared per-pool state -- writable shared state between parent and child process
   An array of these are now stored in the existing ztest_shared_t.

typedef struct ztest_pool_state {
        uint64_t        zps_enospc_count;
        uint64_t        zps_vdev_next_leaf;
        uint64_t        zps_vdev_aux;
        uint64_t        zps_alloc;
        uint64_t        zps_space;
        uint64_t        zps_splits;
        uint64_t        zps_mirrors;
        uint64_t        zps_guid;
} ztest_pool_state_t;

Still TBD

1. Add new tests for importing/exporting and doing some in parallel
2. Add pool name to some of log output
3. Have option for per-pool log file?

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Performance enhancement (non-breaking change which improves efficiency)
• Code cleanup (non-breaking change which makes code smaller or more readable)
• Breaking change (fix or feature that would cause existing functionality to change)
• Library ABI change (libzfs, libzfs_core, libnvpair, libuutil and libzfsbootenv)
• Documentation (a change to man pages or other documentation)

Checklist:

• I have updated the documentation accordingly.
• I have read the contributing document.
• I have added tests to cover my changes.
• I have run the ZFS Test Suite with this change applied.
• All commit messages are properly formatted and contain Signed-off-by.