[EPIC] Add support for all array expressions

[andygrove]

What is the problem the feature request solves?

We would like to support all array operations natively in Comet.

• array: SELECT array(1, 2, 3)
• array_agg: SELECT array_agg(col) FROM VALUES (1), (2), (1) AS tab(col)
• array_append: SELECT array_append(array('b', 'd', 'c', 'a'), 'd') - feat: support array_append #1072
• array_compact: SELECT array_compact(array(1, 2, 3, null))
• array_contains: SELECT array_contains(array(1, 2, 3), 2) - feat: add support for array_contains expression #1163
• array_distinct: SELECT array_distinct(array(1, 2, 3, null, 3))
• array_except: SELECT array_except(array(1, 2, 3), array(1, 3, 5))
• array_insert: SELECT array_insert(array(1, 2, 3, 4), 5, 5) - feat: support array_insert #1073
• array_intersect: SELECT array_intersect(array(1, 2, 3), array(1, 3, 5)) - Feat: Support array_intersect function #1271
• array_join: SELECT array_join(array('hello', 'world'), ' ') - Feat: Support array_join function #1290
• array_max: SELECT array_max(array(1, 20, null, 3)) - Feat: Add support for array_min, array_max, sort_array, array_zip & array_union #1227
• array_min: SELECT array_min(array(1, 20, null, 3)) - Feat: Add support for array_min, array_max, sort_array, array_zip & array_union #1227
• array_position: SELECT array_position(array(3, 2, 1), 1)
• array_remove: SELECT array_remove(array(1, 2, 3, null, 3), 3) - feat: add support for array_remove expression #1179
• array_repeat: SELECT array_repeat('123', 2)
• array_size: SELECT array_size(array('b', 'd', 'c', 'a'))
• array_sort: SELECT array_sort(array(5, 6, 1), (left, right) -> case when left < right then -1 when left > right then 1 else 0 end)
• array_union: SELECT array_union(array(1, 2, 3), array(1, 3, 5)) - Feat: Add support for array_min, array_max, sort_array, array_zip & array_union #1227
• arrays_overlap: SELECT arrays_overlap(array(1, 2, 3), array(3, 4, 5))
• arrays_zip: SELECT arrays_zip(array(1, 2, 3), array(2, 3, 4)) - Feat: Add support for array_min, array_max, sort_array, array_zip & array_union #1227
• sort_array - Feat: Add support for array_min, array_max, sort_array, array_zip & array_union #1227

Describe the potential solution

No response

Additional context

No response

[SemyonSinchenko]

I would like to work on array_zip

[dharanad]

I would like to work on array_contains followed up by sort_array, array_position, array_max, array_min, array_zip, array_union

[jatin510]

I would like to work on array_remove followed up by array_repeat and array_size.
@andygrove

[jatin510]

@andygrove

 select array(1, 2, 3) ;
24/12/19 00:13:47 WARN CometSparkSessionExtensions$CometExecRule: Comet cannot execute some parts of this plan natively (set spark.comet.explainFallback.enabled=false to disable this logging):
 Project [COMET: Project is not native because the following children are not native (Scan OneRowRelation)]
+-  Scan OneRowRelation [COMET: Scan OneRowRelation is not supported]

[1,2,3]

is array function not supported in comet yet ?
it is marked as done above ?

[dharanad]

@andygrove

 select array(1, 2, 3) ;
24/12/19 00:13:47 WARN CometSparkSessionExtensions$CometExecRule: Comet cannot execute some parts of this plan natively (set spark.comet.explainFallback.enabled=false to disable this logging):
 Project [COMET: Project is not native because the following children are not native (Scan OneRowRelation)]
+-  Scan OneRowRelation [COMET: Scan OneRowRelation is not supported]

[1,2,3]

is array function not supported in comet yet ? it is marked as done above ?

To answer you question, Yes array is support by Comet. But if you can observe the error logs. Scan OneRowRelation is not supported by Comet yet. Since it is child of Project, it wasn't converted to ComerProject. Which is why vanilla Project operator is used instead of CometProject

[kazantsev-maksim]

I would like to work on array_except followed up by array_compact.
@andygrove