[Enhancement]: Consider supporting multiple data storage engines

[aimxhaisse]

Use case and current behavior

There are limitations with using LevelDB:

• it's not possible to spread the database on multiple devices
• it's not possible to open the database read-only while it is locked by a process
• it's in maintenance mode, it won't evolve

Other engines like RocksDb support those modes which would help in some setups. In overall, it would improve the performances of EigenDA and also add more supports in term of tooling (i.e: be able to open the database read-only from the outside for example, to have hot-backups from a sidecar).

Support burst of writes while storing on colder devices

The idea is simple, have 2 devices/disks/volumes attached to the machine/vm/pod running EigenDA:

• a hot NVME device for fast write/bulk writes, the engines writes to it directly, this can be 1TiB for instance
• a colder slower one with larger storage for compacted files: the db engine would compact the hot device blocks to the colder one. This would be performed in background where latency is not an issue.

This would:

• improve performances for large setups: it's unlikely you can get a 20TiB NVME setup today with redundancy
• reduce usage of centralizing cloud solutions (which are the only ones today that can realistically handle EigenDA)
• lower cost to operators (i.e: be able to run it on bare-metal for example)

Hot-snapshot

Some engines support multiple opens to the database if they are read-only (LevelDb doesn't), this means it'd be possible to simply add a hot-snapshot feature with a sidecar process, which could upload the database to some remote storage (s3 or similar) wihout having to start/stop the EigenDA process, and without having to worry about corruptions.

Enhancement

• Introduce an abstraction around K/V storage
• Add a second engine (RocksDB likely a good candidate?)
• Maybe other engines with time

Solution proposal

This could be a one-off opt-in when you spin up EigenDA: you select your engine.

Once selected, you can't move at runtime between an engine to another, there could be guides to do this operations though (i.e: via tooling).

Additional Information

No response