[SPARK-36638][SQL][TEST] Generalize OptimizeSkewedJoin - correctness

[zhengruifeng]

this draft is only used to check correctness of the algorithm in #33893

[zhengruifeng]

this local result should be like:

Query 1: answer = 37714
Query 1: splitting []: 37714
Query 1: splitting [7]: 37714
Query 1: splitting [6]: 37714
Query 1: splitting [6,7]: 37714
Query 1: splitting [3]: 37714
Query 1: splitting [3,7]: 37714
Query 1: splitting [3,6]: 37714
Query 1: splitting [3,6,7]: 37714
Query 1: splitting [0]: 37714
Query 1: splitting [0,7]: 37714
Query 1: splitting [0,6]: 37714
Query 1: splitting [0,6,7]: 37714
Query 1: splitting [0,3]: 37714
Query 1: splitting [0,3,7]: 37714
Query 1: splitting [0,3,6]: 37714
Query 1: splitting [0,3,6,7]: 37714
Query 1: random splits
Query 1: splitting []: 37714
Query 1: splitting [9]: 113142
Query 1: splitting [8]: 155044
Query 1: splitting [8,9]: 465132
Query 1: splitting [7]: 37714
Query 1: splitting [7,9]: 113142
Query 1: splitting [7,8]: 155044
Query 1: splitting [7,8,9]: 465132
Query 1: splitting [6]: 37714
Query 1: splitting [6,9]: 113142
Query 1: splitting [6,8]: 155044
Query 1: splitting [6,8,9]: 465132
Query 1: splitting [6,7]: 37714
[info] *** Test still running after 4 minutes, 58 seconds: suite name: GeneralSkewSuite, test name: Query 1: 10-Table Join. 
Query 1: splitting [6,7,9]: 113142
Query 1: splitting [6,7,8]: 155044
Query 1: splitting [6,7,8,9]: 465132
Query 1: splitting [5]: 37714
Query 1: splitting [5,9]: 113142
Query 1: splitting [5,8]: 155044
Query 1: splitting [5,8,9]: 465132
[info] - Query 1: 10-Table Join (7 minutes, 25 seconds)

Query 2: answer = 324
Query 2: splitting []: 324
Query 2: splitting [6]: 324
Query 2: splitting [2]: 324
Query 2: splitting [2,6]: 324
Query 2: splitting [1]: 324
Query 2: splitting [1,6]: 324
Query 2: splitting [1,2]: 324
Query 2: splitting [1,2,6]: 324
Query 2: random splits
Query 2: splitting []: 324
Query 2: splitting [9]: 14266
Query 2: splitting [8]: 35146
Query 2: splitting [8,9]: 151222
Query 2: splitting [7]: 540
Query 2: splitting [7,9]: 17110
Query 2: splitting [7,8]: 42870
Query 2: splitting [7,8,9]: 179690
Query 2: splitting [6]: 324
Query 2: splitting [6,9]: 14266
Query 2: splitting [6,8]: 35146
Query 2: splitting [6,8,9]: 151222
Query 2: splitting [6,7]: 540
Query 2: splitting [6,7,9]: 17110
Query 2: splitting [6,7,8]: 42870
[info] *** Test still running after 2 minutes, 33 seconds: suite name: GeneralSkewSuite, test name: Query 2: 10-Table Join. 
Query 2: splitting [6,7,8,9]: 179690
Query 2: splitting [5]: 540
Query 2: splitting [5,9]: 26895
Query 2: splitting [5,8]: 56775
Query 2: splitting [5,8,9]: 256425
[info] - Query 2: 10-Table Join (3 minutes, 56 seconds)

Query 3: answer = 38038
Query 3: splitting []: 38038
Query 3: splitting [5]: 38038
Query 3: splitting [4]: 38038
Query 3: splitting [4,5]: 38038
Query 3: splitting [2]: 38038
Query 3: splitting [2,5]: 38038
Query 3: splitting [2,4]: 38038
Query 3: splitting [2,4,5]: 38038
Query 3: random splits
Query 3: splitting []: 38038
Query 3: splitting [9]: 146604
Query 3: splitting [8]: 190190
Query 3: splitting [8,9]: 570570
Query 3: splitting [7]: 40930
Query 3: splitting [7,9]: 158380
Query 3: splitting [7,8]: 204650
Query 3: splitting [7,8,9]: 613950
Query 3: splitting [6]: 38038
Query 3: splitting [6,9]: 146604
Query 3: splitting [6,8]: 190190
Query 3: splitting [6,8,9]: 570570
Query 3: splitting [6,7]: 40930
Query 3: splitting [6,7,9]: 158380
Query 3: splitting [6,7,8]: 204650
Query 3: splitting [6,7,8,9]: 613950
[info] *** Test still running after 3 minutes, 36 seconds: suite name: GeneralSkewSuite, test name: Query 3: 10-Table Join. 
Query 3: splitting [5]: 38038
Query 3: splitting [5,9]: 146604
Query 3: splitting [5,8]: 190190
Query 3: splitting [5,8,9]: 570570
[info] - Query 3: 10-Table Join (4 minutes, 5 seconds)

Query 4: answer = 0
Query 4: splitting []: 0
Query 4: splitting [8]: 0
Query 4: splitting [5]: 0
Query 4: splitting [5,8]: 0
Query 4: splitting [4]: 0
Query 4: splitting [4,8]: 0
Query 4: splitting [4,5]: 0
Query 4: splitting [4,5,8]: 0
Query 4: random splits
Query 4: splitting []: 0
Query 4: splitting [9]: 45784
Query 4: splitting [8]: 0
Query 4: splitting [8,9]: 45784
Query 4: splitting [7]: 0
Query 4: splitting [7,9]: 51080
Query 4: splitting [7,8]: 0
Query 4: splitting [7,8,9]: 51080
Query 4: splitting [6]: 0
Query 4: splitting [6,9]: 45784
Query 4: splitting [6,8]: 0
Query 4: splitting [6,8,9]: 45784
Query 4: splitting [6,7]: 0
Query 4: splitting [6,7,9]: 51080
Query 4: splitting [6,7,8]: 0
Query 4: splitting [6,7,8,9]: 51080
Query 4: splitting [5]: 0
Query 4: splitting [5,9]: 45784
Query 4: splitting [5,8]: 0
Query 4: splitting [5,8,9]: 45784
[info] - Query 4: 10-Table Join (4 minutes, 2 seconds)

Query 5: answer = 364
Query 5: splitting []: 364
Query 5: splitting [2]: 364
Query 5: splitting [1]: 364
Query 5: splitting [1,2]: 364
Query 5: random splits
Query 5: splitting []: 364
Query 5: splitting [9]: 1092
Query 5: splitting [8]: 1820
Query 5: splitting [8,9]: 5460
Query 5: splitting [7]: 1820
Query 5: splitting [7,9]: 5460
Query 5: splitting [7,8]: 9100
Query 5: splitting [7,8,9]: 27300
Query 5: splitting [6]: 1233
Query 5: splitting [6,9]: 3699
Query 5: splitting [6,8]: 6165
Query 5: splitting [6,8,9]: 18495
Query 5: splitting [6,7]: 6165
Query 5: splitting [6,7,9]: 18495
Query 5: splitting [6,7,8]: 30825
Query 5: splitting [6,7,8,9]: 92475
Query 5: splitting [5]: 1083
Query 5: splitting [5,9]: 3249
Query 5: splitting [5,8]: 5415
[info] *** Test still running after 5 minutes, 28 seconds: suite name: GeneralSkewSuite, test name: Query 5: 10-Table Join with Agg. 
Query 5: splitting [5,8,9]: 16245
[info] - Query 5: 10-Table Join with Agg (5 minutes, 50 seconds)

Query 6: answer = 898
Query 6: splitting []: 898
Query 6: splitting [8]: 898
Query 6: splitting [7]: 898
Query 6: splitting [7,8]: 898
Query 6: splitting [6]: 898
Query 6: splitting [6,8]: 898
Query 6: splitting [6,7]: 898
Query 6: splitting [6,7,8]: 898
Query 6: random splits
Query 6: splitting []: 898
Query 6: splitting [9]: 1116
Query 6: splitting [8]: 898
Query 6: splitting [8,9]: 1116
Query 6: splitting [7]: 898
Query 6: splitting [7,9]: 1116
Query 6: splitting [7,8]: 898
Query 6: splitting [7,8,9]: 1116
Query 6: splitting [6]: 898
Query 6: splitting [6,9]: 1116
Query 6: splitting [6,8]: 898
Query 6: splitting [6,8,9]: 1116
Query 6: splitting [6,7]: 898
Query 6: splitting [6,7,9]: 1116
[info] *** Test still running after 4 minutes, 38 seconds: suite name: GeneralSkewSuite, test name: Query 6: 10-Table Join with Agg. 
Query 6: splitting [6,7,8]: 898
Query 6: splitting [6,7,8,9]: 1116
Query 6: splitting [5]: 2758
Query 6: splitting [5,9]: 3318
Query 6: splitting [5,8]: 2758
Query 6: splitting [5,8,9]: 3318
[info] - Query 6: 10-Table Join with Agg (7 minutes, 51 seconds)

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48146/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48146/

[SparkQA]

Test build #143634 has finished for PR 34108 at commit 07588a0.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[github-actions]

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!