WIP: Database upgrades #136
Conversation
src/lib/storage/src/lmdb.rs
Outdated
| * 1024 | ||
| * 1024 | ||
| * 1024; | ||
| let rounded_map_size = (map_size as f64 / page_size::get() as f64).round() as usize; |
There was a problem hiding this comment.
let page_size = page_size::get();
let rounded_map_size = ((map_size as f64 / page_size as f64).round() as usize * page_size) as usize;
|
Yeah its good |
|
can i just fucking ask who we is??
|
First off, chill. |
|
@vortexreanimated lmaooo "we" as in the people who worked for ferrumc, duh? LMDB worked pretty good last time: it has low memory usage, and has good access/write speeds. |
This PR provides many changes to the database system, primarily removing the multiple database options and using only LMDB through the heed crate. We've had great success with this database before and multiple databases has gotten very difficult to manage.
Description
This PR removes the other database options and sets LMDB as the only database backend. We used LMDB before and it worked very well, but when rewriting we tried some other databases and had some initial success but eventually they all had too many drawbacks (redb being unreliable, surrealkv and sled using heaps of memory, rockdb requiring long compilation steps, etc.) so it was decided to only use LMDB since it offered very little drawbacks at the cost of having minimal documentation. So far this has lead to a significant reduction in memory use and no noticeable change in performance, though LMDB does use a memory mapping strategy that causes extremely high virtual memory usage. This doesn't have any real effect since the OS will reclaim it as soon as it needs it and all it really represents is the maximum size of the database, but it make our stats look concerning.
This PR also implements simple caching with the moka crate. Currently nothing is being bottlenecked by IO speeds, but this will alleviate a lot of those issues before they arise. The size of the cache is configurable with in the config file, with higher limits allowing for less disk accesses at the cost of higher memory use. The cache drops entries based on a chunk's size in memory and how long it's been since the entry was accessed. The current default is 20MB with a TTL of 60 seconds, meaning chunks are dropped from the cache if they have been there the longest when the cache exceeds 20MB of memory use, or if a chunk hasn't been accessed for 60 seconds. Increasing the size will allow for more chunks to sit in cache at once, while increasing the TTL will mean chunks can stay in cache longer without being dropped, allowing less frequently accessed chunks to maintain fast access speeds.
Some minor API and documentation changes have been made to the anvil parser as well.
Motivation and Context
The database has been a source of annoyance for a while now, requiring different implementations of features for each database. This should allow for a more unified approach.
Caching was implemented to reduce the performance hit that disk reads was having.
How has this been tested?
No tests for now, but proper database tests will be included in future commits.
Screenshots (if appropriate):
Types of changes
Checklist: