Why's the io_uring code so slow?! 🙂

[JackKelly]

• Flamegraph
• Add some metrics to the code to measure how many operations are in flight.
• Try - as a hack - changing rx.recv to recv_timeout, with a small timeout, to wait for data from rx before submitting to uring.
• Drop ObjectStore & async/await. Use Channels instead. Focus entirely (for now) on io_uring for local file storage. #93. And also only pass in entire vectors of operations. Receiver<Vec<Operation>>. This is important because this will affect the design for the Zarr front-end.

[JackKelly]

I think the root cause is that the SQ only ever has one entry in it at a time. I've tried a few "quick fixes" but none seem to help!

[JackKelly]

Printing a few quick hacky metrics suggests that the loops in move_entries_from_internal_queue_to_uring_sq and move_entries_from_injectors_queue_to_uring_sq behave as expected: they do often inject multiple entries onto the SQ.

Which makes me think that maybe each iteration of our main worker loop "just" take too long, so io_uring empties the SQ long before we have a chance to fill it back up again?

[JackKelly]

I've just done some more experiments with fio. It turns out that fio gets similar performance (~1.8 GiB/s) to LSIO when fio uses direct=0, and a single thread. And the flamegraph for LSIO shows that messing around with the page cache is taking a lot of time.

fio shows that using direct=1 and using multiple worker threads is sufficient to saturate the SSD.

So I should use O_DIRECT #51

[JackKelly]

#51 has considerably sped up io_uring! We're now getting more like 5.1 GiB/s on get_1000_whole_files/uring_get

[JackKelly]

After implementing O_DIRECT, uring is now competitive with object_store (phew!)

I think the next "big things" to improve performance will be:

• Drop ObjectStore & async/await. Use Channels instead. Focus entirely (for now) on io_uring for local file storage. #93 (the flamegraph shows that we're spending a fair chunk of time in tokio-related functions... although maybe that's to be expected)
• Multiple threads (each with a uring) & crossbeam::deque & a pipeline of I/O steps #61
• Submit "get filesize" & "open" entries in parallel #69