Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat: Support arrays_overlap function #1312

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

erenavsarogullari
Copy link
Member

@erenavsarogullari erenavsarogullari commented Jan 20, 2025

Which issue does this PR close?

Related to Epic: #1042
arrays_overlap: select arrays_overlap(array('hello', '-', 'world'), array('hello')) => true
DataFusion' s array_has_any has same behavior with Spark 's arrays_overlap function
Spark: https://docs.databricks.com/en/sql/language-manual/functions/arrays_overlap.html
DataFusion: https://datafusion.apache.org/user-guide/sql/scalar_functions.html#array-has-any

Rationale for this change

Defined under Epic: #1042

What changes are included in this PR?

planner.rs: Maps Spark 's arrays_overlap function to DataFusion array_has_any physical expression from Spark physical expression with return type: DataType::Boolean,
expr.proto: arrays_overlap message has been added,
QueryPlanSerde.scala: arrays_overlap pattern matching case has been added,
CometExpressionSuite.scala: A new UT has been added for arrays_overlap function.

How are these changes tested?

A new UT has been added.

@codecov-commenter
Copy link

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 34.56%. Comparing base (2588e13) to head (32b78fd).
Report is 1 commits behind head on main.

Additional details and impacted files
@@             Coverage Diff              @@
##               main    #1312      +/-   ##
============================================
- Coverage     34.72%   34.56%   -0.16%     
+ Complexity      992      965      -27     
============================================
  Files           117      117              
  Lines         45046    45077      +31     
  Branches       9901     9939      +38     
============================================
- Hits          15640    15581      -59     
- Misses        26253    26332      +79     
- Partials       3153     3164      +11     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants