fix: Pick correct columns in Sort Merge Equijoin

[tglanz]

Which issue does this PR close?

• Closes Panic in Sort Merge Equijoin when tables have different columns count #18804.

Rationale for this change

What changes are included in this PR?

Take correct columns

Are these changes tested?

Yes,

• added rust tests to hit invalid code paths
• sqltests
• fuzz tests enhancement to fuzzify columns count

Fuzz tests are taken from @rluvaton 's #18788 , excluding those this PR doesn't fix:

    fuzz_cases::join_fuzz::test_right_anti_join_1k_binary_filtered
    fuzz_cases::join_fuzz::test_right_anti_join_1k_filtered
    fuzz_cases::join_fuzz::test_right_semi_join_1k_filtered

Are there any user-facing changes?

[rluvaton]

I think this should also be changed to:

Suggested change

	.take(left_columns_length)

[rluvaton]

I was able to reproduce the bug when changing the sort_merge_join.slt to have 3 columns in t2 rather than 2 like in t1 (below is the diff I changed in the sort_merge_join.slt file).

Can you please update the description and update/add tests (I would update the sort_merge_join.slt like in the diff below to make sure all tests are testing that. make sure to add a comment on why 3 columns and t1 2).

Updated the slt file to reproduce the error

diff --git a/datafusion/sqllogictest/test_files/sort_merge_join.slt b/datafusion/sqllogictest/test_files/sort_merge_join.slt
--- a/datafusion/sqllogictest/test_files/sort_merge_join.slt	(revision f3980641660997345af6061dc3b34f365020bd07)
+++ b/datafusion/sqllogictest/test_files/sort_merge_join.slt	(date 1763411346050)
@@ -26,7 +26,7 @@
 CREATE TABLE t1(a text, b int) AS VALUES ('Alice', 50), ('Alice', 100), ('Bob', 1);
 
 statement ok
-CREATE TABLE t2(a text, b int) AS VALUES ('Alice', 2), ('Alice', 1);
+CREATE TABLE t2(a text, b int, c int) AS VALUES ('Alice', 2, 77), ('Alice', 1, 66);
 
 # inner join query plan with join filter
 query TT
@@ -64,83 +64,83 @@
 ----
 
 # left join without join filter
-query TITI rowsort
+query TITII rowsort
 SELECT * FROM t1 LEFT JOIN t2 ON t1.a = t2.a
 ----
-Alice 100 Alice 1
-Alice 100 Alice 2
-Alice 50 Alice 1
-Alice 50 Alice 2
-Bob 1 NULL NULL
+Alice 100 Alice 1 66
+Alice 100 Alice 2 77
+Alice 50 Alice 1 66
+Alice 50 Alice 2 77
+Bob 1 NULL NULL NULL
 
 # left join with join filter
-query TITI rowsort
+query TITII rowsort
 SELECT * FROM t1 LEFT JOIN t2 ON t1.a = t2.a AND t2.b * 50 <= t1.b
 ----
-Alice 100 Alice 1
-Alice 100 Alice 2
-Alice 50 Alice 1
-Bob 1 NULL NULL
+Alice 100 Alice 1 66
+Alice 100 Alice 2 77
+Alice 50 Alice 1 66
+Bob 1 NULL NULL NULL
 
-query TITI rowsort
+query TITII rowsort
 SELECT * FROM t1 LEFT JOIN t2 ON t1.a = t2.a AND t2.b < t1.b
 ----
-Alice 100 Alice 1
-Alice 100 Alice 2
-Alice 50 Alice 1
-Alice 50 Alice 2
-Bob 1 NULL NULL
+Alice 100 Alice 1 66
+Alice 100 Alice 2 77
+Alice 50 Alice 1 66
+Alice 50 Alice 2 77
+Bob 1 NULL NULL NULL
 
 # right join without join filter
-query TITI rowsort
+query TITII rowsort
 SELECT * FROM t1 RIGHT JOIN t2 ON t1.a = t2.a
 ----
-Alice 100 Alice 1
-Alice 100 Alice 2
-Alice 50 Alice 1
-Alice 50 Alice 2
+Alice 100 Alice 1 66
+Alice 100 Alice 2 77
+Alice 50 Alice 1 66
+Alice 50 Alice 2 77
 
 # right join with join filter
-query TITI rowsort
+query TITII rowsort
 SELECT * FROM t1 RIGHT JOIN t2 ON t1.a = t2.a AND t2.b * 50 <= t1.b
 ----
-Alice 100 Alice 1
-Alice 100 Alice 2
-Alice 50 Alice 1
+Alice 100 Alice 1 66
+Alice 100 Alice 2 77
+Alice 50 Alice 1 66
 
-query TITI rowsort
+query TITII rowsort
 SELECT * FROM t1 RIGHT JOIN t2 ON t1.a = t2.a AND t1.b > t2.b
 ----
-Alice 100 Alice 1
-Alice 100 Alice 2
-Alice 50 Alice 1
-Alice 50 Alice 2
+Alice 100 Alice 1 66
+Alice 100 Alice 2 77
+Alice 50 Alice 1 66
+Alice 50 Alice 2 77
 
 # full join without join filter
-query TITI rowsort
+query TITII rowsort
 SELECT * FROM t1 FULL JOIN t2 ON t1.a = t2.a
 ----
-Alice 100 Alice 1
-Alice 100 Alice 2
-Alice 50 Alice 1
-Alice 50 Alice 2
-Bob 1 NULL NULL
+Alice 100 Alice 1 66
+Alice 100 Alice 2 77
+Alice 50 Alice 1 66
+Alice 50 Alice 2 77
+Bob 1 NULL NULL NULL
 
-query TITI rowsort
+query TITII rowsort
 SELECT * FROM t1 FULL JOIN t2 ON t1.a = t2.a AND t2.b * 50 > t1.b
 ----
-Alice 100 NULL NULL
-Alice 50 Alice 2
-Bob 1 NULL NULL
-NULL NULL Alice 1
+Alice 100 NULL NULL NULL
+Alice 50 Alice 2 77
+Bob 1 NULL NULL NULL
+NULL NULL Alice 1 66
 
-query TITI rowsort
+query TITII rowsort
 SELECT * FROM t1 FULL JOIN t2 ON t1.a = t2.a AND t1.b > t2.b + 50
 ----
-Alice 100 Alice 1
-Alice 100 Alice 2
-Alice 50 NULL NULL
-Bob 1 NULL NULL
+Alice 100 Alice 1 66
+Alice 100 Alice 2 77
+Alice 50 NULL NULL NULL
+Bob 1 NULL NULL NULL
 
 statement ok
 DROP TABLE t1;

[tglanz]

Thanks @rluvaton, Sure

[tglanz]

Fuzz tests are taken from @rluvaton 's #18788 , excluding those this PR doesn't fix:

    fuzz_cases::join_fuzz::test_right_anti_join_1k_binary_filtered
    fuzz_cases::join_fuzz::test_right_anti_join_1k_filtered
    fuzz_cases::join_fuzz::test_right_semi_join_1k_filtered

[rluvaton]

Thank you @tglanz LGTM.

Hope to see you contributing again soon