Node allocator latency_groups causes just single node to be used for writes

[guymguym]

Environment info

• NooBaa Version: 5.15
• Platform: ODF-4.15

Actual behavior

1. Using a PVPool of 2 or 4 we can see that just a single PV writes at a time.
2. This happens due to this code -

   noobaa-core/src/server/node_services/nodes_monitor.js

   Line 3494 in b27e63b

   if (nodes_with_avg_disk_write.length >= config.NODE_ALLOCATOR_NUM_CLUSTERS) {

3. We can workaround this issue by setting the following config which eliminated this code path:

oc set env sts noobaa-core CONFIG_JS_NODE_ALLOCATOR_NUM_CLUSTERS=99999

Expected behavior

1. All PV's should be writing in parallel.

Steps to reproduce

1. Write to a bucket that uses a PVpool.

More information - Screenshots / Logs / Other output

This is a benchmark running before the workaround - in the write stage you can see a single PV active, and in the read stage both are active:

After the workaround both are writing together.