This is a substantial refactoring of the Ra log implementation. All disk formats remain the same - the changes in this PR primarily deal with:
Changes to memtable management and lifetimes
Refinement of Ra server / WAL / Segment writer messages and handling.
Motivation
Previously the WAL process had quite a lot of responsibilities:
* batching incoming writes from all local ra servers (writers).
* Accounting
* Writing entries to disk and fsyncing
* Notifying ra server post fsync
* Creating and updating new memtable and the mem-table lookup tables. (ra_log_open_mem_tables, ra_log_closed_mem_tables).
* Notifying segment writer
As the WAL is the primary contention point of the Ra log system it was clear it would benefit from fewer responsibilities which should increase the performance and scalability of the log implementation.
In addition the ra server wrote all entries first to a local ETS "cache" table, the same entry that later was written to a memtable by the WAL. So each entry was written to two ETS tables (at a minimum, low priority entries may also be written to yet another ETS table).
V2
In the v2 implementation the WAL no longer writes to memtables. Instead of the ra servers maintaining a cache table this table, in effect, becomes the memtable. Only the ra servers write to memtables now.
The WAL's responsibilities have been reduced to:
* batching
* accounting
* writing and syncing
* notifying
Instead of memtables being linked to the life-time of a WAL file and deleted after being flushed to segments they are now maintained indefinitely and instead flushed entries are deleted from the table.
There are many additional changes and improvements and some API simplifictions.
Breaking change:
The rarely used ra:register_external_reader/2 API has been deprecated and will now only read entries from segment files. So in effect it will provide the same behaviour just delayed.
The docs/LOG_V2.md provides further details on the new log design.
Fix typos
wip