Ra log v2

[kjnilsson]

Log v2

This is a substantial refactoring of the Ra log implementation. All disk formats remain the same - the changes in this PR primarily deal with:

• Changes to memtable management and lifetimes
• Refinement of Ra server / WAL / Segment writer messages and handling.

Motivation

Previously the WAL process had quite a lot of responsibilities:

• batching incoming writes from all local ra servers (writers).
• maintaining accounting information to ensure
• Writing entries to disk and fsyncing
• notifying ra server post fsync
• creating and updating new memtable and the mem-table lookup tables. (ra_log_open_mem_tables, ra_log_closed_mem_tables).

As the WAL is the primary contention point of the Ra log system it was clear it would benefit from fewer responsibilities which should increase the performance and scalability of the log implementation.

In addition the ra server wrote all entries first to a local ETS "cache" table, the same entry that later was written to a memtable by the WAL. So each entry was written to two ETS tables (at a minimum, low priority entries may also be written to yet another ETS table).

V2

In the v2 implementation the WAL no longer writes to memtables. Instead of the ra servers maintaining a cache table this table, in effect, becomes the memtable. Only the ra servers write to memtable now.

The WAL's responsibilities have been reduced to:

• batching
• accounting
• writing and syncing
• notifying

Instead of memtables being linked to the life-time of a WAL file and deleted after being flushed to segments they are now maintained indefinitely and instead flushed entries are deleted from the table.

There are many additional changes and improvements and some API simplifictions.

Breaking change:

The rarely used ra:register_external_reader/2 API has been deprecated and will now only read entries from segment files. So in effect it will provide the same behaviour just delayed.

The docs/internals/LOG_V2.md provides further details on the new log design.

[michaelklishin]

We know that this PR makes quorum queue throughput go up by a non-trivial percentage.

Would be interesting to see the effects of this memtable use change on memory consumption. I assume we'd see the same zig-zag-looking pattern but potentially with a different amplitude.