Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: Allow for more RG skipping by rewriting expr in planner #20828

Draft
wants to merge 2 commits into
base: main
Choose a base branch
from
Draft

feat: Allow for more RG skipping by rewriting expr in planner #20828

wants to merge 2 commits into from

Conversation

coastalwhite
Copy link
Collaborator

This is an drafting PR to show the idea of rewriting the predicate expression in the planner to an expression that can be used to skip record batches (row groups in parquet).

At the moment this is probably already close to what the existing implementation can do even with some more to show that it is better (by implementing Expr.is_in).

@coastalwhite
feat: Allow for more RG skipping by rewriting expr in planner
8deaea1 
This is an drafting PR to show the idea of rewriting the predicate expression
in the planner to an expression that can be used to skip record batches (row
groups in parquet).

At the moment this is probably already close to what the existing
implementation can do even with some more to show that it is better (by
implementing `Expr.is_in`).

[skip ci]
@github-actions github-actions bot added enhancement New feature or an improvement of an existing feature python Related to Python Polars rust Related to Rust Polars labels Jan 21, 2025
@coastalwhite
Copy link
Collaborator Author

This is now basically done, only a few more things need to happen:

  • Port changes to the new and old streaming engine
  • Achieve full feature parity with the old statistics evaluator.

@coastalwhite coastalwhite force-pushed the feat/pred-to-skip-batches-expr branch from f395611 to 1716ad7 Compare January 22, 2025 16:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ritchie46 ritchie46 Awaiting requested review from ritchie46 ritchie46 will be requested when the pull request is marked ready for review ritchie46 is a code owner

@c-peters c-peters Awaiting requested review from c-peters c-peters will be requested when the pull request is marked ready for review c-peters is a code owner

@alexander-beedie alexander-beedie Awaiting requested review from alexander-beedie alexander-beedie will be requested when the pull request is marked ready for review alexander-beedie is a code owner

@MarcoGorelli MarcoGorelli Awaiting requested review from MarcoGorelli MarcoGorelli will be requested when the pull request is marked ready for review MarcoGorelli is a code owner

@reswqa reswqa Awaiting requested review from reswqa reswqa will be requested when the pull request is marked ready for review reswqa is a code owner

@orlp orlp Awaiting requested review from orlp orlp will be requested when the pull request is marked ready for review orlp is a code owner

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
enhancement New feature or an improvement of an existing feature python Related to Python Polars rust Related to Rust Polars
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@coastalwhite