Would it be possibe to swap rocksdb for something else? #201

Kixunil · 2019-11-28T10:46:50Z

I found that it takes enormous time to build and strangely, it's now taking me several hours.

It'd be nice if it could be swapped for something else somehow. Even if its slightly slower.

romanz · 2019-11-29T12:56:19Z

RocksDB build time is indeed an issue - maybe we can try LMDB?
It builds significantly faster :)

$ time cargo build --release
   Compiling cc v1.0.47
   Compiling pkg-config v0.3.17
   Compiling libc v0.2.66
   Compiling bitflags v1.2.1
   Compiling lmdb-sys v0.8.0 (/media/roman/data/roman/Code/Rust/lmdb-rs/lmdb-sys)
warning: /media/roman/data/roman/Code/Rust/lmdb-rs/lmdb-sys/lmdb/libraries/liblmdb/mdb.c: In function ‘mdb_env_get_maxkeysize’:
warning: /media/roman/data/roman/Code/Rust/lmdb-rs/lmdb-sys/lmdb/libraries/liblmdb/mdb.c:10033:33: warning: unused parameter ‘env’ [-Wunused-parameter]
warning:  mdb_env_get_maxkeysize(MDB_env *env)
warning:                         ~~~~~~~~~^~~
warning: /media/roman/data/roman/Code/Rust/lmdb-rs/lmdb-sys/lmdb/libraries/liblmdb/mdb.c: In function ‘mdb_cursor_put’:
warning: /media/roman/data/roman/Code/Rust/lmdb-rs/lmdb-sys/lmdb/libraries/liblmdb/mdb.c:6737:9: warning: this statement may fall through [-Wimplicit-fallthrough=]
warning:       if (SIZELEFT(fp) < offset) {
warning:          ^
warning: /media/roman/data/roman/Code/Rust/lmdb-rs/lmdb-sys/lmdb/libraries/liblmdb/mdb.c:6742:5: note: here
warning:      case MDB_CURRENT:
warning:      ^~~~
   Compiling lmdb v0.8.0 (/media/roman/data/roman/Code/Rust/lmdb-rs)
    Finished release [optimized] target(s) in 4.94s

real	0m4.952s
user	0m12.722s
sys	0m0.429s

romanz · 2019-11-29T13:22:23Z

OTOH, there is https://github.com/spacejam/sled which is pure Rust :)

Kixunil · 2019-11-29T15:19:58Z

LMDB sounds good. Sled is beta, so I think we should wait with that.

Hopefully I will find some time to contribute a PR until the end of January.

GeneFerneau · 2021-10-05T20:05:40Z

LMDB sounds good. Sled is beta, so I think we should wait with that.

2021 and sled is still beta

Just my two cents, maybe a pluggable datastore API makes sense (if not too much extra work). What are your thoughts? Is anyone currently working on this?

Kixunil · 2021-10-05T20:14:19Z

Compile time is partially solved by dynamic linking (and sufficient for me so far) but having change-able database could still be interesting. It should be possible using Clean Architecture, not sure how much work.

GeneFerneau · 2021-10-05T20:16:42Z

Not familiar with Clean Architecture (looks it up).

SQLite might be another good alternative

romanz · 2021-10-21T07:39:23Z

https://crates.io/crates/heed (Rust LMDB/MDBX wrapper) is used by meilisearch/meilisearch#230:

$ cargo build --release
    Updating crates.io index
  Downloaded pkg-config v0.3.20
  Downloaded lmdb-rkv-sys v0.11.0
  Downloaded bytemuck_derive v1.0.1
  Downloaded synchronoise v1.0.0
  Downloaded crossbeam-queue v0.1.2
  Downloaded 5 crates (219.0 KB) in 1.07s
   Compiling proc-macro2 v1.0.30
   Compiling unicode-xid v0.2.2
   Compiling syn v1.0.80
   Compiling serde_derive v1.0.130
   Compiling serde v1.0.130
   Compiling libc v0.2.104
   Compiling ryu v1.0.5
   Compiling cfg-if v0.1.10
   Compiling lazy_static v1.4.0
   Compiling pkg-config v0.3.20
   Compiling serde_json v1.0.68
   Compiling cc v1.0.71
   Compiling itoa v0.4.8
   Compiling heed-traits v0.7.0 (/media/roman/3TB/roman/Code/Rust/heed/heed-traits)
   Compiling byteorder v1.4.3
   Compiling once_cell v1.8.0
   Compiling crossbeam-utils v0.6.6
   Compiling crossbeam-queue v0.1.2
   Compiling synchronoise v1.0.0
   Compiling lmdb-rkv-sys v0.11.0
   Compiling quote v1.0.10
   Compiling page_size v0.4.2
   Compiling bytemuck_derive v1.0.1
   Compiling bytemuck v1.7.2
   Compiling bincode v1.3.3
   Compiling heed-types v0.7.2 (/media/roman/3TB/roman/Code/Rust/heed/heed-types)
   Compiling heed v0.10.6 (/media/roman/3TB/roman/Code/Rust/heed/heed)
    Finished release [optimized] target(s) in 13.88s

romanz · 2021-10-21T11:33:06Z

http://www.lmdb.tech/bench/microbench/ looks promising :)

prusnak · 2021-11-16T16:57:32Z

FWIW there are several crates implementing KV storage via LMDB, with different levels of abstraction:

https://crates.io/crates/rkv (10 months ago) (uses lmdb-rkv below)
https://crates.io/crates/lmdb-rkv (1 year ago)
https://crates.io/crates/heed (10 months ago)
https://crates.io/crates/lmdb (3 years ago)
https://crates.io/crates/lmdb-zero (3 years ago)
https://crates.io/crates/lmdb-rs (4 years ago)

bukatea · 2022-07-03T16:46:34Z

Is there any recent update on this? I'd like to use this crate alongside another crate that depends on an incompatible version of rocksdb

Kixunil · 2022-07-04T10:15:51Z

@bukatea it's not simple, so if you're in a hurry you're probably better off resolving the incompatibility.

bukatea · 2022-07-04T15:25:06Z

@Kixunil thx much, happened to find another solution :)

romanz · 2022-07-25T18:14:44Z

https://github.com/cberner/redb can be an option as well.

romanz · 2023-07-29T19:15:19Z

https://twitter.com/sadisticsystems/status/1684906383227961344 -> https://docs.rs/sled/1.0.0-alpha.1/sled

mikedilger · 2024-07-04T08:57:33Z

I've been using LDMB via https://docs.rs/heed/latest/heed/ for about a year in a nostr client and I'm very happy with it. Heed was recommended to me by Howard Chu the LMDB author as the best rust interface to it.

But you should know the downsides. The database doesn't automatically compress itself, and copy on write with multiple transactions can leave data larger than it needs to be. Using the 'mdb_copy -c' command line resolves this, but there is no rust library call to do this compaction (yet, IIRC).

Sled always looked like a cool idea for optimizing high-contention situations, but too experimental.

I haven't used rocksdb yet so I can't say if LMDB seems better or worse to me.

Kixunil · 2024-07-04T17:44:16Z

FYI since opening this issue I'm less enthusiastic about it. A pure Rust impl would be statically linked which has its own downsides. It's cool how much compile time gets reduced when dynamically linking. (I exclusively link dynamically.)

antonilol · 2024-07-05T21:34:37Z

Even when linking dynamically, the library still has to be compiled (not talking about downloading precompiled libraries here, you can statically link them too then). When I compile electrs (statically) after a code change, rocksdb and all the other crates wont have to be recompiled, simply because of caching. I do agree that dynamic linking is good for disk usage on a system by not duplicating libraries in every binary (I use Arch Linux myself!).

LMDB can be dynamically linked too but LMDB compiles much faster (few seconds!) than RocksDB anyway given the amount of code they each have.

Kixunil · 2024-07-07T03:35:16Z

@antonilol I don't compile rocksdb myself, I use the system one. And caching doesn't help when doing Debian packages with deterministic compilation since it's best to build them from scratch to avoid any surprises.

romanz · 2024-12-17T19:28:20Z

https://github.com/fjall-rs/fjall can be also interesting to try:

Fjall is an LSM-based embeddable key-value storage engine written in Rust.
It features:

Thread-safe BTreeMap-like API

100% safe & stable Rust

Range & prefix searching with forward and reverse iteration

Partitions (a.k.a. column families) with cross-partition atomic semantics

Built-in compression (default = LZ4)

Serializable transactions (optional)

Key-value separation for large blob use cases (optional)

Automatic background maintenance

antonilol · 2024-12-26T21:10:02Z

I will add fjall to #1045 and compare them to the other databases already in there.
I did find fjall-rs/fjall#74 however when browsing their issues, I think in electrs it will not deadlock because the example in the issue can only drop the iterator after finishing the write, but this could still cause writes (after indexing a block) and reads (queries using electrum protocol) to have to wait on eachother. (like an RwLock)

romanz added the question Further information is requested label Nov 29, 2019

romanz mentioned this issue Jan 27, 2020

Can't build with rust 1.40.0 #207

Closed

romanz changed the title ~~Would it be possibe to swap rocksdb for somethig else?~~ Would it be possibe to swap rocksdb for something else? Oct 22, 2021

romanz mentioned this issue Nov 14, 2021

incompatibility with the new version of rocksdb #623

Open

jpmckinney mentioned this issue Mar 24, 2023

Reference: Rust libraries open-contracting/cardinal-rs#4

Closed

jpmckinney mentioned this issue May 18, 2023

Add CLI open-contracting/ocds-merge-rs#9

Open

romanz mentioned this issue May 24, 2024

Rocksdb version #1034

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Would it be possibe to swap rocksdb for something else? #201

Would it be possibe to swap rocksdb for something else? #201

Kixunil commented Nov 28, 2019

romanz commented Nov 29, 2019

romanz commented Nov 29, 2019

Kixunil commented Nov 29, 2019

GeneFerneau commented Oct 5, 2021

Kixunil commented Oct 5, 2021

GeneFerneau commented Oct 5, 2021

romanz commented Oct 21, 2021 •

edited

Loading

romanz commented Oct 21, 2021

prusnak commented Nov 16, 2021 •

edited

Loading

bukatea commented Jul 3, 2022

Kixunil commented Jul 4, 2022

bukatea commented Jul 4, 2022

romanz commented Jul 25, 2022

romanz commented Jul 29, 2023

mikedilger commented Jul 4, 2024

Kixunil commented Jul 4, 2024

antonilol commented Jul 5, 2024

Kixunil commented Jul 7, 2024

romanz commented Dec 17, 2024 •

edited

Loading

antonilol commented Dec 26, 2024

Would it be possibe to swap rocksdb for something else? #201

Would it be possibe to swap rocksdb for something else? #201

Comments

Kixunil commented Nov 28, 2019

romanz commented Nov 29, 2019

romanz commented Nov 29, 2019

Kixunil commented Nov 29, 2019

GeneFerneau commented Oct 5, 2021

Kixunil commented Oct 5, 2021

GeneFerneau commented Oct 5, 2021

romanz commented Oct 21, 2021 • edited Loading

romanz commented Oct 21, 2021

prusnak commented Nov 16, 2021 • edited Loading

bukatea commented Jul 3, 2022

Kixunil commented Jul 4, 2022

bukatea commented Jul 4, 2022

romanz commented Jul 25, 2022

romanz commented Jul 29, 2023

mikedilger commented Jul 4, 2024

Kixunil commented Jul 4, 2024

antonilol commented Jul 5, 2024

Kixunil commented Jul 7, 2024

romanz commented Dec 17, 2024 • edited Loading

antonilol commented Dec 26, 2024

romanz commented Oct 21, 2021 •

edited

Loading

prusnak commented Nov 16, 2021 •

edited

Loading

romanz commented Dec 17, 2024 •

edited

Loading