-
Notifications
You must be signed in to change notification settings - Fork 173
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Feat: Support array_join function #1290
base: main
Are you sure you want to change the base?
Conversation
b98415e
to
5d3c94d
Compare
native/proto/src/proto/expr.proto
Outdated
@@ -86,6 +86,7 @@ message Expr { | |||
ArrayInsert array_insert = 59; | |||
BinaryExpr array_contains = 60; | |||
BinaryExpr array_remove = 61; | |||
BinaryExpr array_join = 63; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why not use 62
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I have another PR for array_intersect and it uses 62. I think array_intersect
can go first.
I think the handling of the third argument (nullReplacement) is missing see: https://docs.databricks.com/en/sql/language-manual/functions/array_join.html#arguments. I think this will also require a new expression type in expr.proto |
Thanks - That is right both Spark and DataFusion support optional |
a5e54c3
to
c5f8e7e
Compare
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #1290 +/- ##
============================================
+ Coverage 34.69% 34.71% +0.01%
- Complexity 991 992 +1
============================================
Files 116 117 +1
Lines 44891 45148 +257
Branches 9864 9956 +92
============================================
+ Hits 15574 15671 +97
- Misses 26168 26310 +142
- Partials 3149 3167 +18 ☔ View full report in Codecov by Sentry. |
c5f8e7e
to
895e701
Compare
lgtm 👍 |
@erenavsarogullari could you fix the conflicts on this PR? |
Thanks @andygrove. Sure, i will have a look in today. |
Which issue does this PR close?
Related to Epic: #1042
array_join
:select array_join(array('hello', '-', 'world'), ' ')
=>hello - world
DataFusion' s
array_join
function is alias of array_to_string function:https://datafusion.apache.org/user-guide/sql/scalar_functions.html#array-join
Rationale for this change
Defined under Epic: #1042
What changes are included in this PR?
planner.rs:
Created DataFusionarray_join
physical expression from Spark physical expression with return type:DataType::Utf8
,expr.proto:
array_join
message has been added (array_intersect is 62),QueryPlanSerde.scala:
array_join
pattern matching case has been added,CometExpressionSuite.scala:
A new UT has been added forarray_join
function.How are these changes tested?
A new UT has been added and first query result is as follows: