TidesDB v0.6.0b

[guycipher]

We are getting further and further everyday. Thank you for those whom contributed to TidesDB and or have joined the TidesDB contribution team! This minor has corrected lots of bugs, stabilized lots of functionality and we added automatic partial background merging. Let's keep it up, we are nearing TidesDB 1!

Features

• ACID transactions are atomic, consistent, isolated, and durable. Transactions are tied to their respective column family.
• Concurrent multiple threads can read and write to the storage engine. Column families use a read-write lock thus allowing multiple readers and a single writer per column family. Transactions on commit and rollback block other threads from reading or writing to the column family until the transaction is completed. A transaction in itself is also is thread safe.
• Column Families store data in separate key-value stores. Each column family has their own memtable and sstables.
• Atomic Transactions commit or rollback multiple operations atomically. When a transaction fails, it rolls back all commited operations.
• Cursor iterate over key-value pairs forward and backward.
• WAL write-ahead logging for durability. Column families replay WAL on startup. This reconstructs memtable if the column family did not reach threshold prior to shutdown.
• Multithreaded Compaction manual multi-threaded paired and merged compaction of sstables. When run for example 10 sstables compacts into 5 as their paired and merged. Each thread is responsible for one pair - you can set the number of threads to use for compaction.
• Background Partial Merge Compaction background merge compaction can be started. If started the system will incrementally merge sstables in the background from oldest to newest. Merges are done every n seconds. Merges are not done in parallel but incrementally.
• Bloom Filters reduce disk reads by reading initial blocks of sstables to check key existence.
• Compression compression is achieved with Snappy, or LZ4, or ZSTD. SStable entries can be compressed as well as WAL entries.
• TTL time-to-live for key-value pairs.
• Configurable column families are configurable with memtable flush threshold, data structure, if skip list max level, if skip list probability, compression, and bloom filters.
• Error Handling API functions return an error code and message.
• Easy API simple and easy to use api.
• Multiple Memtable Data Structures memtable can be a skip list or hash table.
• Multiplatform Linux, MacOS, and Windows support.

Building

Using cmake to build the shared library.

cmake -S . -B build
cmake --build build
cmake --install build

Requirements

You need cmake and a C compiler.
You also require the snappy, lz4, and zstd libraries.

Linux

sudo apt install libzstd-dev
sudo apt install liblz4-dev
sudo apt install libsnappy-dev

MacOS

brew install zstd
brew install lz4
brew install snappy

Windows

Windows using vcpkg

vcpkg install zstd
vcpkg install lz4
vcpkg install snappy

Bindings

Bindings are in the works in various languages.

• tidesdb-cpp
• tidesdb-go
• tidesdb-java
• tidesdb-python
• tidesdb-rust
• tidesdb-lua
• tidesdb-csharp
• tidesdb-scala
• tidesdb-objective-c
• tidesdb-php
• tidesdb-perl
• tidesdb-ruby
• tidesdb-swift
• tidesdb-haskell
• tidesdb-d
• tidesdb-kotlin
• tidesdb-julia
• tidesdb-r
• tidesdb-js
• tidesdb-dart
• tidesdb-nim
• tidesdb-zig
• tidesdb-ocaml
• tidesdb-fsharp
• tidesdb-erlang
• tidesdb-elixir

Discord Community

Join the TidesDB Discord Community to ask questions, work on development, and discuss the future of TidesDB.

Dependencies

• Snappy
• LZ4
• Zstandard

Include

#include <tidesdb.h>

Usage

Each database method returns a tidesdb_err_t* which returns an error code and message. If no error, TidesDB returns NULL.

Example of error structure

typedef struct
{
    int code;
    char *message;
} tidesdb_err_t;

Opening a database

To open a database you pass the path to the database directory and a pointer to the database.

tidesdb_t *tdb = NULL;
tidesdb_err_t *e = tidesdb_open("your_tdb_directory", &tdb);
if (e != NULL)
{
    /* handle error */
    tidesdb_err_free(e);
}

/* close the database */
e = tidesdb_close(tdb);
if (e != NULL)
{
    /* handle error */
    tidesdb_err_free(e);
}

Creating a column family

In order to store data in TidesDB you need a column family. This is by design.

You pass

• the database you want to create the column family in. Must be open
• the name of the column family
• memtable flush threshold in bytes. Example below is 128MB
• skip list max level. Example below is 12 ( only if using TDB_MEMTABLE_SKIP_LIST ) pass TDB_USING_HT_MAX_LEVEL if using TDB_MEMTABLE_HASH_TABLE
• skip list probability. Example below is 0.24 ( only if using TDB_MEMTABLE_SKIP_LIST ) pass TDB_USING_HT_PROBABILITY if using TDB_MEMTABLE_HASH_TABLE
• whether column family sstable data is compressed
• the compression algorithm to use [TDB_NO_COMPRESSION, TDB_COMPRESS_SNAPPY, TDB_COMPRESS_LZ4, TDB_COMPRESS_ZSTD]
• whether to use bloom filters
• what data structure to use for the memtable [TDB_MEMTABLE_SKIP_LIST, TDB_MEMTABLE_HASH_TABLE]

/* create a column family with no compression and no bloom filters (slower reads) */
tidesdb_err_t *e = tidesdb_create_column_family(tdb, "your_column_family", (1024 * 1024) * 128, 12, 0.24f, false, TDB_NO_COMPRESSION, false, TDB_MEMTABLE_SKIP_LIST);
if (e != NULL)
{
    /* handle error */
    tidesdb_err_free(e);
}

Using Snappy compression, bloom filters and hash table memtable.

/* create a column family with compression and bloom filter (the bloom filter provides fast read speed) */
tidesdb_err_t *e = tidesdb_create_column_family(tdb, "your_column_family", (1024 * 1024) * 128, TDB_USING_HT_MAX_LEVEL, TDB_USING_HT_PROBABILITY, true, TDB_COMPRESS_SNAPPY, true, TDB_MEMTABLE_HASH_TABLE);
if (e != NULL)
{
    /* handle error */
    tidesdb_err_free(e);
}

Dropping a column family

/* drop a column family */
tidesdb_err_t *e = tidesdb_drop_column_family(tdb, "your_column_family");
if (e != NULL)
{
    /* handle error */
    tidesdb_err_free(e);
}

Listing column families

/* list column families
 * returns a char* of column family names separated by a newline
*/
char *column_families = tidesdb_list_column_families(tdb);
if (column_families == NULL)
{
    /* handle error */
}

/* in this example we just print and free the column families */
printf("%s\n", column_families);
free(column_families);

Putting a key-value pair

You pass

• the database you want to put the key-value pair in. Must be open
• the column family name you want to put the key-value pair in
• the key
• the key size
• the value
• the value size
• when the key-value pair should expire. If -1 then it never expires.

/* put a key-value pair */
uint8_t key[] = "key";
uint8_t value[] = "value";

tidesdb_err_t *e = tidesdb_put(tdb, "your_column_family", key, sizeof(key), value, sizeof(value), -1);
if (e != NULL)
{
    /* handle error */
    tidesdb_err_free(e);
}

Putting a key-value pair with TTL

/* put a key-value pair with TTL */
uint8_t key[] = "key";
uint8_t value[] = "value";

time_t ttl = time(NULL) + 10; /* 10 seconds */
tidesdb_err_t *e  = tidesdb_put(tdb, "your_column_family", key, sizeof(key), value, sizeof(value), ttl);
if (e != NULL)
{
    /* handle error */
    tidesdb_err_free(e);
}

Getting a key-value pair

You pass

• the database you want to get the key-value pair from. Must be open
• the column family name
• the key
• the key size
• a pointer to the value
• a pointer to the value size

size_t value_len = 0;
uint8_t *value_out = NULL;
uint8_t key[] = "key";

tidesdb_err_t *e = tidesdb_get(tdb, "your_column_family", key, sizeof(key), &value_out, &value_len);
if (e != NULL)
{
    /* handle error */
    tidesdb_err_free(e);
}

Deleting a key-value pair

You pass

• the database you want to delete the key-value pair from. Must be open
• the column family name
• the key
• the key size

uint8_t key[] = "key";

tidesdb_err_t *e = tidesdb_delete(tdb, "your_column_family", key, sizeof(key));
if (e != NULL)
{
    /* handle error */
    tidesdb_err_free(e);
}

Transactions

You can perform a series of operations atomically. This will block other threads from reading or writing to the column family until the transaction is committed or rolled back.

You begin a transaction by calling tidesdb_txn_begin.

You pass

• the database you want to perform the operations in. Must be open
• the transaction pointer
• the column family name you want to perform the operations in

tidesdb_txn_t *transaction;
tidesdb_err_t *e = tidesdb_txn_begin(tdb, &transaction, "your_column_family");
if (e != NULL)
{
    /* handle error */
    tidesdb_err_free(e);
}

Now we can add operations to the transaction.

const uint8_t key[] = "example_key";
const uint8_t value[] = "example_value";
tidesdb_err_t *e = tidesdb_txn_put(transaction, key, sizeof(key), value, sizeof(value), -1); /* you can pass a ttl, similar to put */
if (e != NULL)
{
    /* handle error */
    tidesdb_err_free(e);
}

/* you can add delete operations as well */
e = tidesdb_txn_delete(transaction, key, sizeof(key));
if (e != NULL)
{
    /* handle error */
    tidesdb_err_free(e);
}

/* now we commit */
e = tidesdb_txn_commit(transaction);
if (e != NULL)
{
    /* handle error */
    tidesdb_err_free(e);
}

/* before you free, you can rollback */
tidesdb_err_t *e = tidesdb_txn_rollback(transaction);
if (e != NULL)
{
    /* handle error */
    tidesdb_err_free(e);
}

/* free the transaction */
tidesdb_txn_free(transaction);

Cursors

You can iterate over key-value pairs in a column family.

tidesdb_cursor_t *c;
tidesdb_err_t *e = tidesdb_cursor_init(tdb, "your_column_family", &c);
if (e != NULL)
{
    /* handle error */
    tidesdb_err_free(e);
    return;
}

uint8_t *retrieved_key = NULL;
size_t key_size;
uint8_t *retrieved_value = NULL;
size_t value_size;

/* iterate forward */
while ((e = tidesdb_cursor_next(c)) == NULL)
{
    e = tidesdb_cursor_get(c, &retrieved_key, &key_size, &retrieved_value, &value_size);
    if (e != NULL)
    {
        /* handle error */
        tidesdb_err_free(e);
        break;
    }

    /* use retrieved_key and retrieved_value
     * .. */

    /* free the key and value */
    free(retrieved_key);
    free(retrieved_value);
}

if (e != NULL && e->code != TIDESDB_ERR_AT_END_OF_CURSOR)
{
    /* handle error */
    tidesdb_err_free(e);
}

/* iterate backward */
while ((e = tidesdb_cursor_prev(c)) == NULL)
{
    e = tidesdb_cursor_get(c, &retrieved_key, &key_size, &retrieved_value, &value_size);
    if (e != NULL)
    {
        /* handle error */
        tidesdb_err_free(e);
        break;
    }

    /* use retrieved_key and retrieved_value
     * .. */

    /* free the key and value */
    free(retrieved_key);
    free(retrieved_value);
}

if (e != NULL && e->code != TIDESDB_ERR_AT_START_OF_CURSOR)
{
    /* handle error */
    tidesdb_err_free(e);
}

tidesdb_cursor_free(c);

Compaction

Manual

You can manually compact sstables. This method pairs and merges column family sstables.
Say you have 100, after compaction you will have 50; Always half the amount you had prior. You can set the number of threads to use for compaction.

tidesdb_err_t *e = tidesdb_compact_sstables(tdb, "your_column_family", 10); /* use 10 threads */
if (e != NULL)
{
    /* handle error */
    tidesdb_err_free(e);
}

Automatic / Background Partial Merge Compaction

You can start a background partial merge compaction. This will incrementally merge sstables in the background from oldest to newest. Merges are done every n seconds. Merges are not done in parallel but incrementally. Less blocking than manual compaction.

You pass

• the database you want to start the background partial merge compaction in. Must be open
• the column family name
• the number of seconds to wait before going to next pair and merging
• the minimum number of sstables to trigger a merge

tidesdb_err_t *e = tidesdb_start_partial_merge(tdb, "your_column_family", 10, 10); /* merge a pair every 10 seconds and if there are a minimum 10 sstables */

License

Multiple

Mozilla Public License Version 2.0 (TidesDB)

-- AND --
BSD 3 Clause (Snappy)
BSD 2 (LZ4)
BSD (Zstandard)

What's Changed

• hash_table: fix signedness warnings by @evgenykor in hash_table: fix signedness warnings #218
• • _tidesdb_merge_sstables_w_bloomfilter logical corrections. We put … by @guycipher in - _tidesdb_merge_sstables_w_bloomfilter logical corrections. We put … #219
• bench: corrections on compression-bloom bench and minor reformatting. by @guycipher in bench: corrections on compression-bloom bench and minor reformatting. #220
• • tidesdb_create_column_family logical correction to check memtable d… by @guycipher in - tidesdb_create_column_family logical correction to check memtable d… #221
• • tidesdb_delete since we are already putting a tombstone, no point t… by @guycipher in - tidesdb_delete since we are already putting a tombstone, no point t… #222
• issue optimization: only merge and compact smaller sstables #223 implementation by @guycipher in issue #223 implementation #224
• TDB_MERGE_MULTIPLIER adjustment and minor comment by @guycipher in TDB_MERGE_MULTIPLIER adjustment and minor comment #225
• readme: not on compaction by @guycipher in readme: not on compaction #226
• cmake: fix pthread linkage by @evgenykor in cmake: fix pthread linkage #227
• tidesdb_api: fix directory param to be const char by @evgenykor in tidesdb_api: fix directory param to be const char #228
• Decided the multiplier logic did not make too much sense. I don't th… by @guycipher in Decided the multiplier logic did not make too much sense. I don't th… #229
• README: add lua image by @evgenykor in README: add lua image #232
• Minor enhancements and corrections by @guycipher in Minor enhancements and corrections #233
• bench should only include tidesdb.h which includes compat.h by @guycipher in bench should only include tidesdb.h which includes compat.h #234
• minor comment, read me corrections and couple potential memory leak c… by @guycipher in minor comment, read me corrections and couple potential memory leak c… #235
• tidesdb: tidesdb_create_column_family we should be checking if column family exists earlier in the method by @guycipher in tidesdb: tidesdb_create_column_family we should be checking if column family exists earlier in the method #236
• tidesdb: I've added some definitions to assist user with some default values, say using a hash table for memtable, or a good default max level and prob for skip list TDB_USING_HT_MAX_LEVEL, TDB_USING_HT_PROBABILITY, TDB_DEFAULT_SKIP_LIST_MAX_LEVEL, TDB_DEFAULT_SKIP_LIST_PROBABILITY by @guycipher in tidesdb: I've added some definitions to assist user with some default values, say using a hash table for memtable, or a good default max level and prob for skip list TDB_USING_HT_MAX_LEVEL, TDB_USING_HT_PROBABILITY, TDB_DEFAULT_SKIP_LIST_MAX_LEVEL, TDB_DEFAULT_SKIP_LIST_PROBABILITY #237
• cmake: debug flags for windows if TIDESDB_WITH_SANITIZER on by @guycipher in cmake: debug flags for windows if TIDESDB_WITH_SANITIZER on #238
• cmake-compat: modify cmake as on Windows using CLion you make encount… by @guycipher in cmake-compat: modify cmake as on Windows using CLion you make encount… #239
• removed unused functions, introduced partial merge compaction (automatic as opposed to manual and is optional).. by @guycipher in removed unused functions, introduced partial merge compaction (automatic as opposed to manual and is optional).. #242
• Minor to v0.6.0b, so update cmake and vcpkg.json, rid all cmake build warnings. Correction for tidesdb_partial_merge… by @guycipher in Minor to v0.6.0b, so update cmake and vcpkg.json, rid all cmake build warnings. Correction for _tidesdb_partial_merge_… #243

New Contributors

• @evgenykor made their first contribution in hash_table: fix signedness warnings #218

Full Changelog: v0.5.0b...v0.6.0b

---

This discussion was created from the release v0.6.0b.

[imprvhub]

Keep it up team!. this is a strong step towards v1.0!

[guycipher]

We are getting there slowly but surely!