diff --git a/README.md b/README.md index b5ad277..0d6c4aa 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,7 @@ ### The Raft Consensus Algorithm in Zig This repository houses `zaft` - a [Raft Consensus Algorithm](https://raft.github.io/) library implemented in Zig. It provides the building blocks -for building distributed systems requiring consensus among replicated state machines, like databases. +for creating distributed systems requiring consensus among replicated state machines, like databases. ![tang demo](tang.gif) @@ -46,27 +46,21 @@ const zaft = @import("zaft"); ## Usage +This section will show you how to integrate `zaft` into your system step-by-step. If you want to take look at a fully working example, +check out the [kv_store](./examples/kv_store) - in-memory, replicated key-value store based on `zaft`. + > [!IMPORTANT] > This usage tutorial assumes some familiarity with the Raft Consensus Algorithm. If not, I highly advise you to at least skim through -> the [Raft paper](https://raft.github.io/raft.pdf) and familiarize yourself with how the algorithm works. Don't worry, it's a very well -> written paper! +> the [Raft paper](https://raft.github.io/raft.pdf). Don't worry, it's a short and very well written paper! -Firstly, initialise the `Raft` struct +Firstly, initialise the `Raft` struct: ```zig -// we'll get through to UserData and Entry later +// we'll get to UserData and Entry in a second const Raft = @import("zaft").Raft(UserData, Entry); -const config = Raft.Config{ -}; -const callbacks = Raft.Callbacks{ - // .. -}; -const initial_state = Raft.InitialState{ - // ... -}; - -const raft = Raft.init(config, initial_state, callbacks) +const raft = Raft.init(config, initial_state, callbacks); +defer raft.deinit(); ``` `Raft.init` takes three arguments: @@ -85,21 +79,20 @@ const config = Raft.Config{ * `callbacks` - `Raft` will call this function to perform various actions: ```zig +// makeRPC is used to send Raft messages to other nodes +// this function should be non-blocking, that is, not wait for the response fn makeRPC(ud: *UserData, id: u32, rpc: Raft.RPC) !void { - // makeRPC is used to send Raft messages to other nodes - // this function should be non-blocking, that is, not wait for the response const address = ud.node_addresse[id]; // it's your responsibility to serialize the message, consider using e.g. std.json const msg: []u8 = serialize(rpc); try ud.client.send(address, msg); } +// Entry can be whatever you want +// in this callback the entry should be applied to the state machine +// applying an entry must be deterministic! That is, after applying the +// same entries in the same order, the state machine must be in the same state fn applyEntry(ud: *UserData, entry: Entry) !void { - // Entry can be whatever you want - // in this callback the entry should be applied to the state machine - // applying an entry must be deterministic! That is, after applying the - // same entries in the same order, the state machine must be in the same state - // let's assume that is some kind of key-value store switch(entry) { .add => |add| try ud.store.add(add.key, add.value), @@ -107,33 +100,33 @@ fn applyEntry(ud: *UserData, entry: Entry) !void { } } +// this function needs to persist a new log entry fn logAppend(ud: *UserData, log_entry: Raft.LogEntry) !void { - // this function needs to persist a new log entry try ud.database.appendEntry(log_entry); } +// this function needs to pop the last log entry from the persistent storage fn logPop(ud: *UserData) !Raft.LogEntry { - // this function needs to pop the last log entry from the persistent storage const log_entry = try ud.database.popEntry(); return log_entry; } +// this function needs to persist current_term fn persistCurrentTerm(ud: *UserData, current_term: u32) !void { - // this function needs to persist current_term try ud.database.persistCurrentTerm(current_term); } - +// this function needs to persist voted_for fn persistVotedFor(ud: *UserData, voted_for: ?u32) !void { - // this function needs to persist voted_for try ud.database.persistVotedFor(voted_for); } ``` > [!WARNING] -> Notice that all of the callbacks can return an error for the sake of convinience. +> Notice that all of the callbacks can return an error (mostly for the sake of convinience). +> > Error returned from `makeRPC` will be ignored, the RPC will be simply retried after -> appropriate timeou. Error returned from other function, as of now, will result in panic. +> appropriate timeout. Errors returned from other function, as of now, will result in panic. ```zig // pointer to user_data will be passed as a first argument to all of the callbacks @@ -168,6 +161,43 @@ const initial_state = Raft.InitialState { }; ``` +The `Raft` struct needs to be periodically ticked in order to trigger timeouts and other necessary actions. You can use a separate thread to do that, or +built your app based on an event loop like [libexev](https://github.com/mitchellh/libxev) with its `xev.Timer`. + +```zig +const tick_after = raft.tick(); +// tick_after is a number of milliseconds after which raft should be ticked again +``` + +For instance, [kv_store](./examples/kv_store/src/ticker.zig) uses a separate thread exclusively to tick the `Raft` struct. + +> [!WARNING] +> The `Raft` struct is *not* thread-safe. Use appropriate synchronization means to makes sure it is not accessed simultaneously by many threads, +> like `std.Thread.Mutex`. + +Next, messages from other Raft nodes need to be feed to local `Raft` struct by calling: + +```zig +// you will need to receive and deserialize the messages from other peers +const msg: Raft.RPC = try recv_msg(); +raft.handleRPC(msg); +``` + +Lastly, entries can be appended to `Raft`s log by calling: + +```zig +const entry: Entry = ... +const idx = try raft.appendEntry(entry); +``` + +It will return an index of the new entry. According to the Raft algorithm, you application should block on client request +until the entry has been applied. You can use e.g. `std.Thread.Condition` and call its `notify` function in the `applyEntry` callback. +You can check whether entry was applied by using `raft.checkIfApplied(idx)`. Take a look at how [kv_store](./examples/kv_store/src/main.zig) does this. +`appendEntry` function will return error if the node is not a leader. In such case, you should redirect the client request to the leader node. +You can chack which node is the leader by using `raft.getCurrentLeader()`. You can also check if the node is a leader proactively by calling +`raft.checkifLeader()`. +## Next steps +1. Siema