Fix auto date histogram rounding assertion bug #17023
Merged
+157
−22
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
bug
|
❕ Gradle check result for 0ecdf31: UNSTABLE Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure. |
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #17023 +/- ##
============================================
+ Coverage 72.34% 72.39% +0.05%
- Complexity 65481 65585 +104
============================================
Files 5300 5305 +5
Lines 304330 304622 +292
Branches 44141 44186 +45
============================================
+ Hits 220158 220523 +365
+ Misses 66093 66061 -32
+ Partials 18079 18038 -41 ☔ View full report in Codecov by Sentry. |
finnegancarroll
requested review from
anasalkouz,
andrross,
ashking94,
Bukhtawar,
CEHENKLE,
dblock,
dbwiddis,
gbbafna,
jed326,
kotwanikunal,
mch2,
msfroh,
nknize,
owaiskazi19,
reta,
Rishikesh1159,
sachinpkale,
saratvemulapalli,
shwetathareja,
sohami,
VachaShah,
jainankitk and
linuxpi
as code owners
|
@bowenlan-amzn can you take a look when you have a chance? Thank you! |
|
I remember the optimization will not be applied if the aggregation defined a timezone, so this bug is kind of a surprise. The block of timezone is inside getInterval OpenSearch/server/src/main/java/org/opensearch/common/Rounding.java Lines 1385 to 1388 in 1e49aa8 The timezone is part of the Rounding object, so we need to have Rounding first then check the timezone. However, in auto datehistogram, to get the rounding, we are updating prepared rounding also which leading to this bug. I recommend we add a simple timezone check right before It purely takes in a Rounding (for autodatehistogram, this can just be the first Rounding in RouningInfos, since every Rounding would have the same timezone information) and do the check OpenSearch/server/src/main/java/org/opensearch/common/Rounding.java Lines 1385 to 1388 in 1e49aa8 If the check doesn't pass, we don't even bother to go inside getRounding method. The check you added here (not shrink the rounding) is still meaningful, agree it should be ever increasing, but not shrink depending on the next segment processed. |
|
The Jenkins job url is https://build.ci.opensearch.org/job/benchmark-pull-request/2222/ . Final results will be published once the job is completed. |
jainankitk
reviewed
Jan 28, 2025
jainankitk
approved these changes
Jan 28, 2025
|
Retrying |
opensearch-trigger-bot bot
pushed a commit
that referenced
this pull request
Jan 28, 2025
* Add comments explanations for auto date histo increaseRoundingIfNeeded. Signed-off-by: Finn Carroll <[email protected]> * Add testFilterRewriteWithTZRoundingRangeAssert() to reproduce auto date histo assertion bug per #16932 Signed-off-by: Finn Carroll <[email protected]> * Fix #16932. Ensure optimized path can only increase preparedRounding of agg. Signed-off-by: Finn Carroll <[email protected]> * Spotless apply Signed-off-by: Finn Carroll <[email protected]> * Fast fail filter rewrite opt in data histo aggs for non UTC timezones Signed-off-by: Finn Carroll <[email protected]> * Remove redundant UTC check from getInterval(). Signed-off-by: Finn Carroll <[email protected]> * Save a call to prepareRounding if roundingIdx is unchanged. Signed-off-by: Finn Carroll <[email protected]> * Spotless apply Signed-off-by: Finn Carroll <[email protected]> * Changelog Signed-off-by: Finn Carroll <[email protected]> * Add ZoneId getter for date histo filter rewrite canOptimize check. Signed-off-by: Finn Carroll <[email protected]> * Spotless apply Signed-off-by: Finn Carroll <[email protected]> * Disable ff optimzation for composite agg in canOptimize. Signed-off-by: Finn Carroll <[email protected]> * Spotless apply Signed-off-by: Finn Carroll <[email protected]> * Handle utc timezone check Signed-off-by: bowenlan-amzn <[email protected]> * Remove redundant timeZone getter. Signed-off-by: Finn Carroll <[email protected]> * Simplify ff prepared rounding check. Signed-off-by: Finn Carroll <[email protected]> --------- Signed-off-by: Finn Carroll <[email protected]> Signed-off-by: bowenlan-amzn <[email protected]> Co-authored-by: bowenlan-amzn <[email protected]> (cherry picked from commit de59264) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Benchmark ResultsBenchmark Results for Job: https://build.ci.opensearch.org/job/benchmark-pull-request/2222/
|
Benchmark Baseline Comparison ResultsBenchmark Results for Job: https://build.ci.opensearch.org/job/benchmark-compare/37/
|
|
@finnegancarroll @jainankitk asc sort, range and keyword-in-range queries seem to have regressed. |
I think I ran the wrong workload here. This PR should have been id_15 since that's the lucene 10 snapshot and baseline is main. Running id_5 workload again on 2.x backport: #17175 (comment) |
finnegancarroll
pushed a commit
to finnegancarroll/OpenSearch
that referenced
this pull request
Jan 29, 2025
) * Add comments explanations for auto date histo increaseRoundingIfNeeded. Signed-off-by: Finn Carroll <[email protected]> * Add testFilterRewriteWithTZRoundingRangeAssert() to reproduce auto date histo assertion bug per opensearch-project#16932 Signed-off-by: Finn Carroll <[email protected]> * Fix opensearch-project#16932. Ensure optimized path can only increase preparedRounding of agg. Signed-off-by: Finn Carroll <[email protected]> * Spotless apply Signed-off-by: Finn Carroll <[email protected]> * Fast fail filter rewrite opt in data histo aggs for non UTC timezones Signed-off-by: Finn Carroll <[email protected]> * Remove redundant UTC check from getInterval(). Signed-off-by: Finn Carroll <[email protected]> * Save a call to prepareRounding if roundingIdx is unchanged. Signed-off-by: Finn Carroll <[email protected]> * Spotless apply Signed-off-by: Finn Carroll <[email protected]> * Changelog Signed-off-by: Finn Carroll <[email protected]> * Add ZoneId getter for date histo filter rewrite canOptimize check. Signed-off-by: Finn Carroll <[email protected]> * Spotless apply Signed-off-by: Finn Carroll <[email protected]> * Disable ff optimzation for composite agg in canOptimize. Signed-off-by: Finn Carroll <[email protected]> * Spotless apply Signed-off-by: Finn Carroll <[email protected]> * Handle utc timezone check Signed-off-by: bowenlan-amzn <[email protected]> * Remove redundant timeZone getter. Signed-off-by: Finn Carroll <[email protected]> * Simplify ff prepared rounding check. Signed-off-by: Finn Carroll <[email protected]> --------- Signed-off-by: Finn Carroll <[email protected]> Signed-off-by: bowenlan-amzn <[email protected]> Co-authored-by: bowenlan-amzn <[email protected]> (cherry picked from commit de59264) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
finnegancarroll
pushed a commit
to finnegancarroll/OpenSearch
that referenced
this pull request
Jan 29, 2025
) * Add comments explanations for auto date histo increaseRoundingIfNeeded. Signed-off-by: Finn Carroll <[email protected]> * Add testFilterRewriteWithTZRoundingRangeAssert() to reproduce auto date histo assertion bug per opensearch-project#16932 Signed-off-by: Finn Carroll <[email protected]> * Fix opensearch-project#16932. Ensure optimized path can only increase preparedRounding of agg. Signed-off-by: Finn Carroll <[email protected]> * Spotless apply Signed-off-by: Finn Carroll <[email protected]> * Fast fail filter rewrite opt in data histo aggs for non UTC timezones Signed-off-by: Finn Carroll <[email protected]> * Remove redundant UTC check from getInterval(). Signed-off-by: Finn Carroll <[email protected]> * Save a call to prepareRounding if roundingIdx is unchanged. Signed-off-by: Finn Carroll <[email protected]> * Spotless apply Signed-off-by: Finn Carroll <[email protected]> * Changelog Signed-off-by: Finn Carroll <[email protected]> * Add ZoneId getter for date histo filter rewrite canOptimize check. Signed-off-by: Finn Carroll <[email protected]> * Spotless apply Signed-off-by: Finn Carroll <[email protected]> * Disable ff optimzation for composite agg in canOptimize. Signed-off-by: Finn Carroll <[email protected]> * Spotless apply Signed-off-by: Finn Carroll <[email protected]> * Handle utc timezone check Signed-off-by: bowenlan-amzn <[email protected]> * Remove redundant timeZone getter. Signed-off-by: Finn Carroll <[email protected]> * Simplify ff prepared rounding check. Signed-off-by: Finn Carroll <[email protected]> --------- Signed-off-by: Finn Carroll <[email protected]> Signed-off-by: bowenlan-amzn <[email protected]> Co-authored-by: bowenlan-amzn <[email protected]> (cherry picked from commit de59264) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
3 tasks
|
The local benchmark doesn't replicate this behavior, could be a one-off bad run. Submitting one more confirm. |
|
{"run-benchmark-test": "id_5"} |
|
The Jenkins job url is https://build.ci.opensearch.org/job/benchmark-pull-request/2231/ . Final results will be published once the job is completed. |
Benchmark ResultsBenchmark Results for Job: https://build.ci.opensearch.org/job/benchmark-pull-request/2231/
|
Benchmark Baseline Comparison ResultsBenchmark Results for Job: https://build.ci.opensearch.org/job/benchmark-compare/38/
|
finnegancarroll
pushed a commit
to finnegancarroll/OpenSearch
that referenced
this pull request
Jan 30, 2025
) * Add comments explanations for auto date histo increaseRoundingIfNeeded. Signed-off-by: Finn Carroll <[email protected]> * Add testFilterRewriteWithTZRoundingRangeAssert() to reproduce auto date histo assertion bug per opensearch-project#16932 Signed-off-by: Finn Carroll <[email protected]> * Fix opensearch-project#16932. Ensure optimized path can only increase preparedRounding of agg. Signed-off-by: Finn Carroll <[email protected]> * Spotless apply Signed-off-by: Finn Carroll <[email protected]> * Fast fail filter rewrite opt in data histo aggs for non UTC timezones Signed-off-by: Finn Carroll <[email protected]> * Remove redundant UTC check from getInterval(). Signed-off-by: Finn Carroll <[email protected]> * Save a call to prepareRounding if roundingIdx is unchanged. Signed-off-by: Finn Carroll <[email protected]> * Spotless apply Signed-off-by: Finn Carroll <[email protected]> * Changelog Signed-off-by: Finn Carroll <[email protected]> * Add ZoneId getter for date histo filter rewrite canOptimize check. Signed-off-by: Finn Carroll <[email protected]> * Spotless apply Signed-off-by: Finn Carroll <[email protected]> * Disable ff optimzation for composite agg in canOptimize. Signed-off-by: Finn Carroll <[email protected]> * Spotless apply Signed-off-by: Finn Carroll <[email protected]> * Handle utc timezone check Signed-off-by: bowenlan-amzn <[email protected]> * Remove redundant timeZone getter. Signed-off-by: Finn Carroll <[email protected]> * Simplify ff prepared rounding check. Signed-off-by: Finn Carroll <[email protected]> --------- Signed-off-by: Finn Carroll <[email protected]> Signed-off-by: bowenlan-amzn <[email protected]> Co-authored-by: bowenlan-amzn <[email protected]> (cherry picked from commit de59264) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
|
results look fine now. |
finnegancarroll
pushed a commit
to finnegancarroll/OpenSearch
that referenced
this pull request
Jan 30, 2025
) * Add comments explanations for auto date histo increaseRoundingIfNeeded. Signed-off-by: Finn Carroll <[email protected]> * Add testFilterRewriteWithTZRoundingRangeAssert() to reproduce auto date histo assertion bug per opensearch-project#16932 Signed-off-by: Finn Carroll <[email protected]> * Fix opensearch-project#16932. Ensure optimized path can only increase preparedRounding of agg. Signed-off-by: Finn Carroll <[email protected]> * Spotless apply Signed-off-by: Finn Carroll <[email protected]> * Fast fail filter rewrite opt in data histo aggs for non UTC timezones Signed-off-by: Finn Carroll <[email protected]> * Remove redundant UTC check from getInterval(). Signed-off-by: Finn Carroll <[email protected]> * Save a call to prepareRounding if roundingIdx is unchanged. Signed-off-by: Finn Carroll <[email protected]> * Spotless apply Signed-off-by: Finn Carroll <[email protected]> * Changelog Signed-off-by: Finn Carroll <[email protected]> * Add ZoneId getter for date histo filter rewrite canOptimize check. Signed-off-by: Finn Carroll <[email protected]> * Spotless apply Signed-off-by: Finn Carroll <[email protected]> * Disable ff optimzation for composite agg in canOptimize. Signed-off-by: Finn Carroll <[email protected]> * Spotless apply Signed-off-by: Finn Carroll <[email protected]> * Handle utc timezone check Signed-off-by: bowenlan-amzn <[email protected]> * Remove redundant timeZone getter. Signed-off-by: Finn Carroll <[email protected]> * Simplify ff prepared rounding check. Signed-off-by: Finn Carroll <[email protected]> --------- Signed-off-by: Finn Carroll <[email protected]> Signed-off-by: bowenlan-amzn <[email protected]> Co-authored-by: bowenlan-amzn <[email protected]> (cherry picked from commit de59264) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
finnegancarroll
pushed a commit
to finnegancarroll/OpenSearch
that referenced
this pull request
Jan 30, 2025
) * Add comments explanations for auto date histo increaseRoundingIfNeeded. Signed-off-by: Finn Carroll <[email protected]> * Add testFilterRewriteWithTZRoundingRangeAssert() to reproduce auto date histo assertion bug per opensearch-project#16932 Signed-off-by: Finn Carroll <[email protected]> * Fix opensearch-project#16932. Ensure optimized path can only increase preparedRounding of agg. Signed-off-by: Finn Carroll <[email protected]> * Spotless apply Signed-off-by: Finn Carroll <[email protected]> * Fast fail filter rewrite opt in data histo aggs for non UTC timezones Signed-off-by: Finn Carroll <[email protected]> * Remove redundant UTC check from getInterval(). Signed-off-by: Finn Carroll <[email protected]> * Save a call to prepareRounding if roundingIdx is unchanged. Signed-off-by: Finn Carroll <[email protected]> * Spotless apply Signed-off-by: Finn Carroll <[email protected]> * Changelog Signed-off-by: Finn Carroll <[email protected]> * Add ZoneId getter for date histo filter rewrite canOptimize check. Signed-off-by: Finn Carroll <[email protected]> * Spotless apply Signed-off-by: Finn Carroll <[email protected]> * Disable ff optimzation for composite agg in canOptimize. Signed-off-by: Finn Carroll <[email protected]> * Spotless apply Signed-off-by: Finn Carroll <[email protected]> * Handle utc timezone check Signed-off-by: bowenlan-amzn <[email protected]> * Remove redundant timeZone getter. Signed-off-by: Finn Carroll <[email protected]> * Simplify ff prepared rounding check. Signed-off-by: Finn Carroll <[email protected]> --------- Signed-off-by: Finn Carroll <[email protected]> Signed-off-by: bowenlan-amzn <[email protected]> Co-authored-by: bowenlan-amzn <[email protected]> (cherry picked from commit de59264) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
3 tasks
jainankitk
pushed a commit
that referenced
this pull request
Jan 30, 2025
* Fix auto date histogram rounding assertion bug (#17023) * Add comments explanations for auto date histo increaseRoundingIfNeeded. Signed-off-by: Finn Carroll <[email protected]> * Add testFilterRewriteWithTZRoundingRangeAssert() to reproduce auto date histo assertion bug per #16932 Signed-off-by: Finn Carroll <[email protected]> * Fix #16932. Ensure optimized path can only increase preparedRounding of agg. Signed-off-by: Finn Carroll <[email protected]> * Spotless apply Signed-off-by: Finn Carroll <[email protected]> * Fast fail filter rewrite opt in data histo aggs for non UTC timezones Signed-off-by: Finn Carroll <[email protected]> * Remove redundant UTC check from getInterval(). Signed-off-by: Finn Carroll <[email protected]> * Save a call to prepareRounding if roundingIdx is unchanged. Signed-off-by: Finn Carroll <[email protected]> * Spotless apply Signed-off-by: Finn Carroll <[email protected]> * Changelog Signed-off-by: Finn Carroll <[email protected]> * Add ZoneId getter for date histo filter rewrite canOptimize check. Signed-off-by: Finn Carroll <[email protected]> * Spotless apply Signed-off-by: Finn Carroll <[email protected]> * Disable ff optimzation for composite agg in canOptimize. Signed-off-by: Finn Carroll <[email protected]> * Spotless apply Signed-off-by: Finn Carroll <[email protected]> * Handle utc timezone check Signed-off-by: bowenlan-amzn <[email protected]> * Remove redundant timeZone getter. Signed-off-by: Finn Carroll <[email protected]> * Simplify ff prepared rounding check. Signed-off-by: Finn Carroll <[email protected]> --------- Signed-off-by: Finn Carroll <[email protected]> Signed-off-by: bowenlan-amzn <[email protected]> Co-authored-by: bowenlan-amzn <[email protected]> (cherry picked from commit de59264) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * Remove breaking abstract isUTC() getter from Rounding.java. Signed-off-by: Finn Carroll <[email protected]> * Remove unused ZoneId getter. Signed-off-by: Finn Carroll <[email protected]> --------- Signed-off-by: Finn Carroll <[email protected]> Signed-off-by: bowenlan-amzn <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: bowenlan-amzn <[email protected]>
opensearch-trigger-bot bot
pushed a commit
that referenced
this pull request
Jan 30, 2025
* Fix auto date histogram rounding assertion bug (#17023) * Add comments explanations for auto date histo increaseRoundingIfNeeded. Signed-off-by: Finn Carroll <[email protected]> * Add testFilterRewriteWithTZRoundingRangeAssert() to reproduce auto date histo assertion bug per #16932 Signed-off-by: Finn Carroll <[email protected]> * Fix #16932. Ensure optimized path can only increase preparedRounding of agg. Signed-off-by: Finn Carroll <[email protected]> * Spotless apply Signed-off-by: Finn Carroll <[email protected]> * Fast fail filter rewrite opt in data histo aggs for non UTC timezones Signed-off-by: Finn Carroll <[email protected]> * Remove redundant UTC check from getInterval(). Signed-off-by: Finn Carroll <[email protected]> * Save a call to prepareRounding if roundingIdx is unchanged. Signed-off-by: Finn Carroll <[email protected]> * Spotless apply Signed-off-by: Finn Carroll <[email protected]> * Changelog Signed-off-by: Finn Carroll <[email protected]> * Add ZoneId getter for date histo filter rewrite canOptimize check. Signed-off-by: Finn Carroll <[email protected]> * Spotless apply Signed-off-by: Finn Carroll <[email protected]> * Disable ff optimzation for composite agg in canOptimize. Signed-off-by: Finn Carroll <[email protected]> * Spotless apply Signed-off-by: Finn Carroll <[email protected]> * Handle utc timezone check Signed-off-by: bowenlan-amzn <[email protected]> * Remove redundant timeZone getter. Signed-off-by: Finn Carroll <[email protected]> * Simplify ff prepared rounding check. Signed-off-by: Finn Carroll <[email protected]> --------- Signed-off-by: Finn Carroll <[email protected]> Signed-off-by: bowenlan-amzn <[email protected]> Co-authored-by: bowenlan-amzn <[email protected]> (cherry picked from commit de59264) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * Remove breaking abstract isUTC() getter from Rounding.java. Signed-off-by: Finn Carroll <[email protected]> * Remove unused ZoneId getter. Signed-off-by: Finn Carroll <[email protected]> --------- Signed-off-by: Finn Carroll <[email protected]> Signed-off-by: bowenlan-amzn <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: bowenlan-amzn <[email protected]> (cherry picked from commit a79c6e8) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
jainankitk
pushed a commit
that referenced
this pull request
Jan 30, 2025
* Fix auto date histogram rounding assertion bug (#17023) * Add comments explanations for auto date histo increaseRoundingIfNeeded. Signed-off-by: Finn Carroll <[email protected]> * Add testFilterRewriteWithTZRoundingRangeAssert() to reproduce auto date histo assertion bug per #16932 Signed-off-by: Finn Carroll <[email protected]> * Fix #16932. Ensure optimized path can only increase preparedRounding of agg. Signed-off-by: Finn Carroll <[email protected]> * Spotless apply Signed-off-by: Finn Carroll <[email protected]> * Fast fail filter rewrite opt in data histo aggs for non UTC timezones Signed-off-by: Finn Carroll <[email protected]> * Remove redundant UTC check from getInterval(). Signed-off-by: Finn Carroll <[email protected]> * Save a call to prepareRounding if roundingIdx is unchanged. Signed-off-by: Finn Carroll <[email protected]> * Spotless apply Signed-off-by: Finn Carroll <[email protected]> * Changelog Signed-off-by: Finn Carroll <[email protected]> * Add ZoneId getter for date histo filter rewrite canOptimize check. Signed-off-by: Finn Carroll <[email protected]> * Spotless apply Signed-off-by: Finn Carroll <[email protected]> * Disable ff optimzation for composite agg in canOptimize. Signed-off-by: Finn Carroll <[email protected]> * Spotless apply Signed-off-by: Finn Carroll <[email protected]> * Handle utc timezone check Signed-off-by: bowenlan-amzn <[email protected]> * Remove redundant timeZone getter. Signed-off-by: Finn Carroll <[email protected]> * Simplify ff prepared rounding check. Signed-off-by: Finn Carroll <[email protected]> --------- Signed-off-by: Finn Carroll <[email protected]> Signed-off-by: bowenlan-amzn <[email protected]> Co-authored-by: bowenlan-amzn <[email protected]> (cherry picked from commit de59264) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * Remove breaking abstract isUTC() getter from Rounding.java. Signed-off-by: Finn Carroll <[email protected]> * Remove unused ZoneId getter. Signed-off-by: Finn Carroll <[email protected]> --------- Signed-off-by: Finn Carroll <[email protected]> Signed-off-by: bowenlan-amzn <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: bowenlan-amzn <[email protected]>
andrross
pushed a commit
that referenced
this pull request
Jan 31, 2025
#17211) * Fix auto date histogram rounding assertion bug (#17023) * Add comments explanations for auto date histo increaseRoundingIfNeeded. * Add testFilterRewriteWithTZRoundingRangeAssert() to reproduce auto date histo assertion bug per #16932 * Fix #16932. Ensure optimized path can only increase preparedRounding of agg. * Spotless apply * Fast fail filter rewrite opt in data histo aggs for non UTC timezones * Remove redundant UTC check from getInterval(). * Save a call to prepareRounding if roundingIdx is unchanged. * Spotless apply * Changelog * Add ZoneId getter for date histo filter rewrite canOptimize check. * Spotless apply * Disable ff optimzation for composite agg in canOptimize. * Spotless apply * Handle utc timezone check * Remove redundant timeZone getter. * Simplify ff prepared rounding check. --------- (cherry picked from commit de59264) * Remove breaking abstract isUTC() getter from Rounding.java. * Remove unused ZoneId getter. --------- (cherry picked from commit a79c6e8) Signed-off-by: Finn Carroll <[email protected]> Signed-off-by: bowenlan-amzn <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: bowenlan-amzn <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
The auto date histogram agg contains an optimization which skips traditional doc collection and instead examines the "pre-aggregated" doc counts contained in the BKD tree of each segment. Several conditions need to be true before this optimization can execute for a given segment. One such condition is we must be able to determine a set of ranges (or rounding) for the segment under consideration before optimizing.
Normally ranges for the auto date histogram aggregation are updated as needed over the course of collecting documents but the filter rewrite optimization will update the
preparedRoundingof the agg in accordance with the min & max values of the segment under consideration ahead of time since it skips regular doc collection.As a result, it is possible for our
preparedRoundingto shrink as the rounding built from the segment could easily be smaller than the rounding previously used by our shard level aggregator.This usually does not pose a problem as the
preparedRoundingwill be updated accordingly when we collect our next document, or reduce our shard level aggs into a single top level agg.The specific case where this becomes problematic is when our
preparedRoundingis delegating to a "bounded" structure. When we prepare a rounding we do so for the min & max epoch time of our shard since this allows us to optimize the structure we delegate rounding to.For some ranges of epoch time and time zones rounding will be little more than a modulo operation. However if our min & max epoch time crosses "transitions" such as daylight savings we may want to delegate rounding to a linked list or array structure to quickly lookup these transitions. This is why the specific occurrence of this bug linked in the initial issue only appears when
"time_zone":"America/New_York".The combination of delegating rounding to these strictly bounded structures and the filter rewrite optimization "replaying" our previous bucket keys fails an assertion within our
preparedRoundingas our previous bucket keys are not guaranteed to fit within the strict bounds of the rounding prepared for the current segment being collected.The changes in this PR resolve this by ensuring the filter rewrite optimization only ever increases the granularity of our
preparedRounding.Related Issues
Resolves #16932
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.