Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rework attributes to be instance based #38

Merged
merged 5 commits into from
Apr 10, 2024
Merged

Rework attributes to be instance based #38

merged 5 commits into from
Apr 10, 2024

Conversation

halgari
Copy link
Contributor

@halgari halgari commented Apr 10, 2024

Pretty massive overhaul of the attribute system to handle #37. More code was deleted here than added. And the net effect is that we're back to 400k datoms/sec insert on my machine.

What this PR does:

  • Moves most of the dictionary lookups AttrId -> Attribute, AttrId -> ValueSerializer, etc. into inline arrays.
    • Since attributes are always in the range of 1 to 2^16 we can pre-allocate slots for all the attributes.
    • Likewise, when mapping attributes to DB Ids the chances of us having more than X database connections open at once is very rare, so we can pre-allocate those caches as well. X is currently set to 8.
  • Transactions now work on pooled writers instead of by having each insert be a IWriteDatom class. This means that when writing data we never allocate a datom, just write the data straight to a buffer and then the transaction system streams it to disk.
    • Some improvement in the future here for constructing buffers of data during Transactor construction and never copying them until we write the transaction to storage
  • Reworks all the test models to use the new format
  • Reduces the number of virtual calls in a lot of places, we now use Attribute directly insteadof via IAttribute.
  • Cleans up the parameters used when creating Attribute<T>, and uses enums were useful.
  • Moves AttributeId from a ulong to a ushort. The values will never be higher than 2^16 anyway, but we had them as ulongs as technically they are EntityIds.
  • Cleans up the word salad that was the separation of models and attributes. Now we define attributes in a static class, with the models defined as subclasses. So we can write File.Path, File.Size and read the data via File.Model

@halgari halgari merged commit 21115f8 into main Apr 10, 2024
6 checks passed
@halgari halgari deleted the instance-attributes branch April 10, 2024 20:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant