-
Notifications
You must be signed in to change notification settings - Fork 676
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
SOLR-17447 : Support to early terminate a search based on maxHits per collector. #2960
base: main
Are you sure you want to change the base?
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Overall this makes sense to me. Thanks for the nice contribution! I'd prefer if @gus-asf could take a look at some aspects since he worked on cpuAllowed recently. BTW it's clear you need to run ./gradlew tidy
I suggest renaming the param "maxHitsPerShard" to simply "maxHits" or "maxHitsTerminateEarly" and document that it's per-shard and best-effort; that more hits may ultimately be detected in aggregate. But maybe others disagree.
It'd be good to consider interference with other features. Maybe it works with cursorMark? Does it count after PostFilter (e.g. CollapseQParser) or before?
Renamed to maxHits
|
Adds the capability to limit hits per shard. "maxHitsPerShard" request parameter controls how many hits the searcher should run over per shard. Once the searcher runs over the specfied number of documents, it will terminate the search with EarlyTerminatingCollectorException. This will be indicated by a new response header "terminatedEarly" also the "partialResults" will indicate that the results are partial. This parameter is supported in MT mode as well. Though there are other mechanisms to control runaway queries with CPU usage limits and time limits, this is simpler for certain use cases esp in case high recall queries and rerank use cases.
rename parameter to maxHits
Description
Adds the capability to early terminate a search based on maxHits parameter provided
https://issues.apache.org/jira/browse/SOLR-17447
Solution
"maxHitsPerShard" request parameter controls how many hits the searcher should go over per shard. Once the searcher runs over the specfied number of documents, it will terminate the search with EarlyTerminatingCollectorException. This will be indicated by a new response header "terminatedEarly" also the "partialResults" will indicate that the results are partial. This parameter is supported in MT mode as well.
Though there are other mechanisms to control runaway queries with CPU usage limits and time limits, this is simpler for certain use cases esp in case high recall queries and rerank use cases.
Lucene currently supports this feature with the EarlyTerminatingCollector. There was some code in SOLR as well to support the collector, but looks like it was not completely wired up.
Tests
Ran tests against a local solr instance in MT and single threaded mode
Checklist
Please review the following and check all that apply:
main
branch../gradlew check
.