Skip to content

[SPARK-37652][SQL]Add test for optimize skewed join through union#34908

Closed
mcdull-zhang wants to merge 1 commit into
apache:masterfrom
mcdull-zhang:skewed_union
Closed

[SPARK-37652][SQL]Add test for optimize skewed join through union#34908
mcdull-zhang wants to merge 1 commit into
apache:masterfrom
mcdull-zhang:skewed_union

Conversation

@mcdull-zhang
Copy link
Copy Markdown
Contributor

@mcdull-zhang mcdull-zhang commented Dec 15, 2021

What changes were proposed in this pull request?

#34974, solved most scenarios of data skew in union.
add test for it.

Why are the changes needed?

Added tests for the following scenarios:

scenes 1

Union
    SMJ
        ShuffleQueryStage
        ShuffleQueryStage
    SMJ
        ShuffleQueryStage
        ShuffleQueryStage

scenes 2

Union
    SMJ
        ShuffleQueryStage
        ShuffleQueryStage
    HashAggregate

scenes 3: not yet supported, SMJ-3 will introduce a new shuffle, so SMJ-1 cannot be optimized

Union
    SMJ-1
        ShuffleQueryStage
        ShuffleQueryStage
    SMJ-2
       SMJ-3
         ShuffleQueryStage
         ShuffleQueryStage
       HashAggregate

Does this PR introduce any user-facing change?

No

How was this patch tested?

Pass the added test

@github-actions github-actions Bot added the SQL label Dec 15, 2021
@AmplabJenkins
Copy link
Copy Markdown

Can one of the admins verify this patch?

@mcdull-zhang
Copy link
Copy Markdown
Contributor Author

cc @cloud-fan

@zhengruifeng
Copy link
Copy Markdown
Contributor

I think #34974 can handle this case.

@cloud-fan
Copy link
Copy Markdown
Contributor

Yea, I think this is already supported.

@ulysses-you
Copy link
Copy Markdown
Contributor

Although we have supported it, I think it's still good to add some test. @mcdull-zhang can you rebase this PR only for the test ?

@mcdull-zhang mcdull-zhang reopened this Feb 9, 2022
@mcdull-zhang mcdull-zhang changed the title [SPARK-37652][SQL]Support optimize skewed join through union [SPARK-37652][SQL]Add test for optimize skewed join through union Feb 9, 2022
@mcdull-zhang
Copy link
Copy Markdown
Contributor Author

@ulysses-you @cloud-fan Please take a look, does the test code make sense? If it doesn't make sense, I'll turn off the pr.

@cloud-fan
Copy link
Copy Markdown
Contributor

thanks, merging to master!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants