[BugFix] isLocalBucketShuffleJoin return wrong result

[satanson]

Why I'm doing:

No data sent from the right side of Bucket Shuffle Right Join Build (ExchangeSink)

1. First, the Bucket seqs 3, 5, 7, 8, 9 of the left table in the Bucket shuffle join were pruned.

2. Generally, when bucket seqs are pruned, the instance id on the receiver side in the channel map is recorded as -1 to indicate no data is being sent. However, for right outer join, the NULL values from the build side need to be output, so for such cases, any valid instance id must be assigned to the pruned bucket seq.

for (int32_t channel_id : _channel_indices) {
    if (_channels[channel_id]->get_fragment_instance_id().lo == -1) {
        // dest bucket is not used, continue
        continue;
    }

    for (int32_t i = 0; i < _num_shuffles_per_channel; ++i) {
        int shuffle_id = channel_id * _num_shuffles_per_channel + i;
        int driver_sequence = _driver_sequence_per_shuffle[shuffle_id];

        size_t from = _channel_row_idx_start_points[shuffle_id];
        size_t size = _channel_row_idx_start_points[shuffle_id + 1] - from;
        if (size == 0) {
            // no data for this channel, continue;
            continue;
        }

        RETURN_IF_ERROR(_channels[channel_id]->add_rows_selective(send_chunk, 
driver_sequence,
                                                                  
_row_indexes.data(), from, size, state));
    }
}

3. In practice, all the join column data on the right-hand side is NULL, and no valid instance id is assigned in the channel map, resulting in data being filtered.
   

2. The issue where NULL values on the HashJoin Build side need to be output, and assigning any instance id to the pruned bucketSeqs

1. Cases where NULL values on the build side need to be output:

   a. Right join, full join, Right Anti JOIN

   b. Null-safe-eq join: <=>. It seems this is not currently considered in the logic, further confirmation is required.

2. Determine whether NULL values on the build side need to be output by isRightOrFullBucketShuffleFragment.

3. isRightOrFullBucketShuffleFragment is expected to be true, but is actually false.

if (isRightOrFullBucketShuffleFragment && colocatedAssignment.isAllScanNodesAssigned()) {
    int bucketNum = colocatedAssignment.bucketNum;

    for (int bucketSeq = 0; bucketSeq < bucketNum; ++bucketSeq) {
        if (!bucketSeqToWorkerId.containsKey(bucketSeq)) {
            long workerId = workerProvider.selectNextWorker();
            bucketSeqToWorkerId.put(bucketSeq, workerId);
        }
        if (!bucketSeqToScanRange.containsKey(bucketSeq)) {
            bucketSeqToScanRange.put(bucketSeq, Maps.newHashMap());
            bucketSeqToScanRange.get(bucketSeq).put(scanNode.getId().asInt(), 
Lists.newArrayList());
        }
    }
}

Top-Down Determination of isRightOrFullBucketShuffle: depends on first visited

private boolean isLocalBucketShuffleJoin(PlanNode root) {
    if (root instanceof ExchangeNode) {
        return false;
    }

    if (root instanceof JoinNode) {
        JoinNode joinNode = (JoinNode) root;
        if (joinNode.isLocalHashBucket()) {
            isRightOrFullBucketShuffle = 
joinNode.getJoinOp().isFullOuterJoin() || joinNode.getJoinOp().isRightJoin();
            return true;
        }
    }

    boolean childHasBucketShuffle = false;
    for (PlanNode child : root.getChildren()) {
        childHasBucketShuffle |= isLocalBucketShuffleJoin(child);
    }

    return childHasBucketShuffle;
}

In the erroneous plan, there are three JOINS, and the above bucket shuffle left join is the first visited, directly determining isRightOrFullBucketShuffle = false.

What I'm doing:

Fix function isLocalBucketShuffleJoin to return correct result.

What type of PR is this:

• BugFix
• Feature
• Enhancement
• Refactor
• UT
• Doc
• Tool

Does this PR entail a change in behavior?

• Yes, this PR will result in a change in behavior.
• No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

• Interface/UI changes: syntax, type conversion, expression evaluation, display information
• Parameter changes: default values, similar parameters but with different default values
• Policy changes: use new policy to replace old one, functionality automatically enabled
• Feature removed
• Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

• I have added test cases for my bug fix or my new feature
• This pr needs user documentation (for new or modified features or behaviors)
  • I have added documentation for my new feature or new function
• This is a backport pr

Bugfix cherry-pick branch check:

• I have checked the version labels which the pr will be auto-backported to the target branch
  • 3.3
  • 3.2
  • 3.1
  • 3.0
  • 2.5

[ZiheLiu]

Where willisRightOrFullBucketShuffle be set to true, maybe as follows?

Suggested change

	hasBucketShuffle = joinNode.getJoinOp().isFullOuterJoin() || joinNode.getJoinOp().isRightJoin();
	hasBucketShuffle = true;
	isRightOrFullBucketShuffle ||= joinNode.getJoinOp().isFullOuterJoin() || joinNode.getJoinOp().isRightJoin();

[github-actions]

[FE Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

[sonarcloud]

Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

[github-actions]

[Java-Extensions Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

[github-actions]

[BE Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

[github-actions]

@Mergifyio backport branch-3.3

[github-actions]

@Mergifyio backport branch-3.2

[github-actions]

@Mergifyio backport branch-3.1

[github-actions]

@Mergifyio backport branch-3.0

[github-actions]

@Mergifyio backport branch-2.5

[mergify]

backport branch-3.3

✅ Backports have been created

• #51977 [BugFix] isLocalBucketShuffleJoin return wrong result (backport #51954) has been created for branch branch-3.3

[mergify]

backport branch-3.2

✅ Backports have been created

• #51978 [BugFix] isLocalBucketShuffleJoin return wrong result (backport #51954) has been created for branch branch-3.2

[mergify]

backport branch-3.1

✅ Backports have been created

• #51979 [BugFix] isLocalBucketShuffleJoin return wrong result (backport #51954) has been created for branch branch-3.1 but encountered conflicts

[mergify]

backport branch-3.0

✅ Backports have been created

• #51980 [BugFix] isLocalBucketShuffleJoin return wrong result (backport #51954) has been created for branch branch-3.0 but encountered conflicts

[mergify]

backport branch-2.5

✅ Backports have been created

• #51981 [BugFix] isLocalBucketShuffleJoin return wrong result (backport #51954) has been created for branch branch-2.5 but encountered conflicts