Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ra log v2 #475

Merged
merged 2 commits into from
Nov 12, 2024
Merged

Ra log v2 #475

merged 2 commits into from
Nov 12, 2024

Commits on Nov 11, 2024

  1. Ra log v2

    This is a substantial refactoring of the Ra log implementation. All disk formats remain the same - the changes in this PR primarily deal with:
    
    Changes to memtable management and lifetimes
    Refinement of Ra server / WAL / Segment writer messages and handling.
    Motivation
    Previously the WAL process had quite a lot of responsibilities:
    
    * batching incoming writes from all local ra servers (writers).
    * Accounting
    * Writing entries to disk and fsyncing
    * Notifying ra server post fsync
    * Creating and updating new memtable and the mem-table lookup tables. (ra_log_open_mem_tables, ra_log_closed_mem_tables).
    * Notifying segment writer
    
    As the WAL is the primary contention point of the Ra log system it was clear it would benefit from fewer responsibilities which should increase the performance and scalability of the log implementation.
    
    In addition the ra server wrote all entries first to a local ETS "cache" table, the same entry that later was written to a memtable by the WAL. So each entry was written to two ETS tables (at a minimum, low priority entries may also be written to yet another ETS table).
    
    V2
    
    In the v2 implementation the WAL no longer writes to memtables. Instead of the ra servers maintaining a cache table this table, in effect, becomes the memtable. Only the ra servers write to memtables now.
    
    The WAL's responsibilities have been reduced to:
    
    * batching
    * accounting
    * writing and syncing
    * notifying
    
    Instead of memtables being linked to the life-time of a WAL file and deleted after being flushed to segments they are now maintained indefinitely and instead flushed entries are deleted from the table.
    
    There are many additional changes and improvements and some API simplifictions.
    
    Breaking change:
    
    The rarely used ra:register_external_reader/2 API has been deprecated and will now only read entries from segment files. So in effect it will provide the same behaviour just delayed.
    
    The docs/LOG_V2.md provides further details on the new log design.
    
    Fix typos
    
    wip
    kjnilsson committed Nov 11, 2024
    Configuration menu
    Copy the full SHA
    24c7c00 View commit details
    Browse the repository at this point in the history

Commits on Nov 12, 2024

  1. additional logging

    kjnilsson committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    822be50 View commit details
    Browse the repository at this point in the history