feat: support EXISTS/IN subquery expressions outside filters via mark joins

[fornwall]

Which issue does this PR close?

No tracking issue in this repository. Upstream tracks the same gap as apache#23022 (under the subquery epic apache#5483); this PR is fork-internal.

Rationale for this change

DecorrelatePredicateSubquery only rewrites IN/EXISTS subqueries that appear in a Filter predicate (or join condition). A subquery expression anywhere else — the select list, a CASE branch, an ORDER BY key — survives optimization untouched and the physical planner fails with a raw internal error:

SELECT EXISTS(SELECT 1);
-- This feature is not implemented: Physical plan does not support logical
-- expression Exists(Exists { subquery: <subquery>, negated: false })

DataFusion already has exactly the right tool: the LeftMark join, whose boolean mark column is precisely EXISTS semantics, already used by this rule for subqueries inside filter disjunctions.

What changes are included in this PR?

• DecorrelatePredicateSubquery now also matches Projection and Sort nodes whose expressions contain EXISTS/IN subqueries, turns each subquery into a LeftMark join on the node's input via the existing mark_join path, and replaces the expression with the mark column (aliased to keep output column names; a projection restores the schema above a Sort).
• EXISTS is always rewritten — correlated or not, negated or not — since the mark column is exactly its two-valued semantics. Non-equality correlations work too (PullUpCorrelatedExpr pulls them into the join filter).
• IN is only rewritten when exact: the mark column is two-valued while IN is three-valued. In a filter that difference is unobservable (a NULL predicate drops rows just like FALSE — unchanged behavior), but in a value position the rewrite is applied only when neither the left-hand expression nor the subquery output can produce a NULL. Nullable cases keep failing in the physical planner rather than silently returning FALSE where NULL is required.
• The plan invariant that rejected In/Exists subqueries in Sort nodes during analysis now allows them (they previously failed with a planning error before the rewrite could run).
• build_join_column_index in sort-pushdown now handles RightMark joins (a mark join under a Sort gets side-swapped into one by hash-join optimization, which previously hit unreachable!).

Are these changes tested?

• New sqllogictest cases in subquery.slt: uncorrelated/empty/negated EXISTS in the select list, correlated EXISTS (equality and non-equality correlation), EXISTS inside CASE, non-nullable IN/NOT IN in the select list, and EXISTS/IN as sort keys (the IN-as-sort-key case exercises the RightMark sort-pushdown fix).
• The existing subquery.slt case asserting the old analyzer rejection of ORDER BY ... IN (subquery) now asserts the physical planner's not-implemented error for the nullable-IN case.
• cargo fmt, workspace cargo clippy --all-targets --all-features -- -D warnings, the full sqllogictest suite (494/494 files), and the extended workspace test suite (9901 tests) all pass.

Are there any user-facing changes?

EXISTS(...) and non-nullable IN (subquery) expressions now work in select lists, CASE expressions, and ORDER BY keys instead of erroring. Nullable IN subqueries in those positions still return the not-implemented error (never a wrong three-valued result).

🤖 Generated with Claude Code

https://claude.ai/code/session_01WS4N9hte6f1W5r6EuDzobY