Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Dynamic shards #2632

Open
ivakegg opened this issue Nov 18, 2024 · 1 comment · May be fixed by #2691
Open

Dynamic shards #2632

ivakegg opened this issue Nov 18, 2024 · 1 comment · May be fixed by #2691
Assignees
@lbschanno

Comments

@ivakegg
Copy link
Collaborator

ivakegg commented Nov 18, 2024

We need a mechanism such that when the size of the data in a day's worth of shards gets too large, then ingest will automatically start using the next set of shards. For example if the number of shards is currently 10, then once some configurable threshold is met then shards 10-19 will be used instead of 0-9.
@ivakegg ivakegg self-assigned this Nov 18, 2024
@lbschanno lbschanno assigned lbschanno and unassigned ivakegg Jan 6, 2025
@ivakegg
Copy link
Collaborator Author

ivakegg commented Jan 6, 2025 •
edited
Loading

The idea is to be able to configure the ShardIdFactory with classes that implement some interface which can be used to override the shard id for an event. The method needs to be passed in the event (of course), the shard id that the factory would have calculated, and the max number of shards for the event.

lbschanno added a commit that referenced this issue Jan 17, 2025
@lbschanno
Add support for dynamic shard creation
873c155 
Add the ability for define custom ShardIdGenerators that can be added
to ShardIdFactory and used to override shard id generation for any
records considered applicable for the generators.

Closes #2632
@lbschanno lbschanno linked a pull request Jan 17, 2025 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@lbschanno lbschanno

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Feature/dynamic shards NationalSecurityAgency/datawave
2 participants
@ivakegg @lbschanno