Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Demonstrate an adaptive hierarchical min reduction #523

Draft
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

frankmcsherry
Copy link
Member

This PR demonstrates through examples/min.rs a hierarchical aggregation that adapts to the group size. For an input collection of (key, val) records, it produces (key, val, lvl) results with the properties that:

  1. The value of lvl is one of two values, those which sandwich the logarithm of the count for the key (under your favorite base).
  2. Each update to the input results in at most a bounded number of updates to the output, generally by at most the base used above for the logarithm (e.g. that many values may get bumped to a higher level, or drawn back down to a lower level).

The algorithm is based on the idea that one can think of putting values into bins at each of various levels, defined by (key, lvl, hash(val) % base.pow(lvl)). We'll use (key, count) to mark various bins as "full", and ask each (key, val) to iteratively discover which level they should be at (starting at zero). With some care in each of these steps, they seem to have the property (modulo bugs, fundamental errors) that updates to the input result in bounded work to update the output.

cc: @petrosagg @ggevay

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant