New crate: apply a user-supplied function to a Stream of buffers

[JackKelly]

Each item will be a

struct<T> Item {
    buffer: &[u8], 
    user_data: T, 
} 

User data is required to allow users to identify the buffers after they've been compressed because the order might change

Update: Framed might already do this: https://docs.rs/async-codec/latest/async_codec/

Related

• Consider mixing Tokio & Rayon #25
• Try interleaving compute with IO #37
• Think about how to keep using object_store to read data from cloud object storage, perhaps using an adaptor which converts object_store to a channel of data chunks.

todo

• Use a flat crate structure so this git repo can store multiple (interconnected) crates in a single workspace #94

[JackKelly]

But remember that, per chunk, we want all processing to happen in one go, so that the data for that chunk is still in cpu cache.

Instead of a compression crate, Maybe make a "datastream" crate. The user provides a processing function (which might include compression). And the crate runs that function in a rayon thread pool. The crate could still provide hooks for common compression libs. Or maybe that should be another crate

[JackKelly]

And maybe make a wrapper which can pull object_store data into a Stream, so folks can use this parallel-processing crate with object_store.

UPDATE: See #107

[JackKelly]

This issue is superseded by #106