[fix](topn) Fix topn distinct union output mapping#64900
Open
morrySnow wants to merge 1 commit into
Open
Conversation
Issue Number: close #xxx Related PR: apache#27628 Problem Summary: PushDownTopNDistinctThroughUnion used the rewritten child plan output to map UNION output slots when pushing TopN distinct into UNION branches. For UNION DISTINCT queries with duplicate projection columns and window output, the child plan output can be shorter than the UNION regular child output, causing IndexOutOfBoundsException during rewrite. Use the UNION regular child output for slot remapping and add focused FE and regression coverage for duplicate output with window plus outer ORDER BY LIMIT. None - Test: Regression test / Unit Test - ./run-fe-ut.sh --run org.apache.doris.nereids.rules.rewrite.PushDownTopNDistinctThroughUnionTest - ./run-regression-test.sh --run -d nereids_rules_p0/push_down_top_n -s push_down_top_n_distinct_through_union -forceGenOut - Behavior changed: No - Does this need documentation: No
morrySnow
requested review from
924060929,
englefly and
starocean999
as code owners
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
/review |
Contributor
Author
|
run buildall |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
I found one correctness gap that should be fixed before merging: the TopN path now uses the UNION regular child outputs, but the parallel LIMIT DISTINCT-through-UNION rewrite still uses post-rewrite child output positions and can hit the same duplicate-output/window failure for LIMIT-only queries.
Critical checkpoint conclusions:
- Goal: the PR fixes the ORDER BY/LIMIT TopN variant, but does not fully accomplish the stated root fix across the parallel LIMIT path.
- Scope/focus: the implementation is small and local; the missing parallel rule update is the main scope gap.
- Concurrency/lifecycle/config/protocol/transactions/persistence: not involved in these changed FE rewrite/test files.
- Parallel paths:
PushDownLimitDistinctThroughUnionis a functionally parallel path and still has the old mapping. - Tests: the new FE and regression coverage exercise the TopN/ORDER BY case; a no-ORDER-BY LIMIT case is still missing.
- Validation: I did not run FE or regression tests because
thirdparty/installed/bin/protocandthirdparty/installed/bin/thriftare missing in this runner.git diff --check FETCH_HEAD..HEADon the changed files passed.
User focus: no additional user-provided review focus was supplied.
Subagent conclusions: optimizer-rewrite proposed OPT-1, which I independently verified and accepted as MAIN-1 for the inline comment. tests-session-config reported no TEST-* candidates. In convergence round 1, after MAIN-1 was accepted and the final comment set was recorded, both live subagents replied NO_NEW_VALUABLE_FINDINGS.
| NamedExpression output = union.getOutputs().get(i); | ||
| replaceMap.put(output, child.getOutput().get(i)); | ||
| for (int j = 0; j < union.getOutputs().size(); ++j) { | ||
| NamedExpression output = union.getOutputs().get(j); |
Contributor
There was a problem hiding this comment.
This fixes the TopN variant, but the parallel LIMIT-only path still has the same output mapping bug. BuildAggForUnion rewrites UNION DISTINCT into a distinct LogicalAggregate over union.withAllQualifier(), and a query with ... LIMIT 10 but no outer ORDER BY reaches PushDownLimitDistinctThroughUnion instead of this TopN rule. That rule still loops over union.getOutputs().size() and reads child.getOutput().get(i), so the duplicate-column/window shape added in this PR can still index past the pruned child output on the LIMIT-only path. Please apply the same union.getRegularChildOutput(childIdx) mapping in PushDownLimitDistinctThroughUnion and add a no-ORDER-BY LIMIT case.
Contributor
TPC-H: Total hot run time: 29453 ms |
Contributor
TPC-DS: Total hot run time: 172573 ms |
Contributor
ClickBench: Total hot run time: 25.13 s |
Contributor
FE UT Coverage ReportIncrement line coverage |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related PR: #27628
Problem Summary: PushDownTopNDistinctThroughUnion used the rewritten child plan output to map UNION output slots when pushing TopN distinct into UNION branches. For UNION DISTINCT queries with duplicate projection columns and window output, the child plan output can be shorter than the UNION regular child output, causing IndexOutOfBoundsException during rewrite. Use the UNION regular child output for slot remapping and add focused FE and regression coverage for duplicate output with window plus outer ORDER BY LIMIT.