HIVE-28572: Support Distribute by and Cluster by clauses in CBO

[kasakrisz]

What changes were proposed in this pull request?

Currently CBO is disabled when a query has DISTRIBUTE BY or CLUSTER BY clauses. This patch lift this limitation by implementing the support of these clauses.

1. Enable these type of conversions from AST, QB to RelNode:

• DISTRIBUTE BY key0..keyn -> HiveSortExchange(distribution=[hash[key0...keyn]], collation=[[]])
• DISTRIBUTE BY key0..keyn SORT BY sortKey0...sortKeyn -> HiveSortExchange(distribution=[hash[key0...keyn]], collation=[[sortKey0...sortKeyn]])
• CLUSTER BY key0..keyn -> HiveSortExchange(distribution=[hash[key0...keyn]], collation=[[key0...keyn]])
  Hence CBO remains enabled

2. At RelNode to AST the conversion HiveSortExchange is converted back to

TOK_DISTRIBUTEBY
  key0
  ..
  keyn
TOK_SORTBY
  sortKey0
  ...
  sortKeyn

3. When CBO return path is enabled distribute keys are set as partition keys to the ReduceSinkDesc.
4. Extend HiveFilterSortTransposeRule to enable pushing through Filter on SortExchange
5. HiveProjectSortExchangeTransposeRule: do not push down Project when it does not project necessary expressions referenced by distribution keys in SortExchange

Why are the changes needed?

CBO provides many optimizations at the logical level. Queries having DISTRIBUTE BY or CLUSTER BY clauses can also benefit from these.

Does this PR introduce any user-facing change?

No but execution plan may have changes.

Is the change a dependency upgrade?

No.

How was this patch tested?

mvn test -Dtest.output.overwrite -Dtest=TestMiniLlapLocalCliDriver -Dqfile=distributeby.q,distributeby_cboret.q -pl itests/qtest -Pitests

[sonarcloud]

Quality Gate passed

Issues
11 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud