chore: rebase Greptime thin fork onto DataFusion 54.1

[discord9]

Which issue does this PR close?

N/A — issues are disabled on this maintenance fork. This replacement is tracked by GreptimeDB PR #8555 and supersedes GreptimeTeam/datafusion#20.

Rationale for this change

GreptimeDB is upgrading from its DataFusion 53.1 maintenance branch to DataFusion 54.1. The first thin-fork proposal (#20) was built on official 54.0.0. Since then, official 54.1.0 shipped 20 backports, including an adapted version of the volatile file-scan projection fix that #20 carried locally.

This replacement starts from the exact official 54.1.0 commit 0d1f2ebe2cc97c91b736bc0a160b5b73cf40437a, drops the now-upstream volatile projection patch, and carries only the correctness fixes still required by GreptimeDB. It keeps the organization branch reviewable instead of making the official 54.1 update appear as part of a PR based on greptimedb-54.0.0.

What changes are included in this PR?

Base:

GreptimeTeam/datafusion:greptimedb-54.1.0
0d1f2ebe2cc97c91b736bc0a160b5b73cf40437a

Retained fixes, replayed with provenance from the validated 54.0 stack:

Source	Correctness behavior retained	Source commit	54.1 commit
GreptimeTeam #5	Preserve mixed aggregate-filter predicate order	7303323513	a60023a3a
GreptimeTeam #16 / Apache apache#23357	Account for grouped AVG/MEDIAN accumulator memory	467e7ec99f	e5cfcc3a8
GreptimeTeam #14 / Apache apache#23271	Count ragged nested-array cardinality recursively	c349e4b388	062d818d9
GreptimeTeam #15 / Apache apache#23306	Bound array_resize by Arrow's maximum array size	9297b40fd4	e4231af92
GreptimeTeam #4 / Apache apache#22759	Preserve finer timestamp precision during coercion	557ddad236	a286efa5d
GreptimeTeam #13	Make fetch-aware statistics safe and apply them to SortPreservingMergeExec	e2be63976c	4b31e11d0
GreptimeTeam #12	Preserve global LIMIT/OFFSET semantics across partitions	d385af852d	33f4161e7
DF54 follow-up	Keep safe TRY_CAST scalar/array overflow behavior nullable	fb7b26d04c	218a14c53
GreptimeTeam #11	Replace unsafe cast unwrapping with exact/range cast preimages	027358559b	887df12d5
DF54 follow-up	Return SQL NULL for exact median(NULL) and fall back from grouped accumulation	0f6dcb128	d450ac6b5
DF54 follow-up	Preserve duplicate Substrait root names with deterministic internal qualifiers	cee35d94c	33e6bb173
Apache apache#23677	Preserve global ORDER BY requirements below ScalarSubqueryExec roots	9eae6da6f	43a6b5750
GreptimeTeam #22	Preserve NULLs when evaluating regex predicates on dictionary values	aa6f9fa05	b1f616b1d
GreptimeTeam #23	Rewrite ordered coarse-to-fine timestamp-widening predicates to source-domain bounds (ceil/floor, aligned and non-aligned)	f2a0b22e4	d75a5582a

Explicitly not replayed:

• adfae3451 — official 54.1 contains the adapted fix as c735a49a3.
• GreptimeTeam backport: Allow dynamic filter pushdown for left join #3 and fix: treat NULL sort merge join filters as unmatched #19 — already covered by the official 54 release line.
• GreptimeTeam fix: report sort preserving merge cardinality effect #18 — no consumer-level optimizer regression demonstrates an impact after the fetch-statistics correction.

The resulting candidate head is d75a5582a7112e11bf2f36449f47fd60ac820a05, exactly 14 commits above the official 54.1 base.

Overflow policy disclosure (GreptimeTeam #23)

The ordered timestamp-widening rewrite introduced in d75a5582a is applied with the same accepted fork policy as the merged DF53 PR #23: the ordered source bound is exactly equivalent only for source values whose widening is representable. Outside that domain a regular CAST can error and TRY_CAST can return NULL, while the rewrite still emits the source-domain predicate. No full-domain equivalence is claimed; the commit message and datafusion/expr-common/src/casts.rs document this trade-off.

Are these changes tested?

Each retained semantic change has focused regression coverage from the original test-first port. The 54.1 replacement additionally requires:

• ancestry and exact commit count from 0d1f2ebe2;
• range-diff against the validated 54.0 stack;
• no thin-fork diff in the official volatile projection implementation, its Cargo files, or its SQL logic test;
• workspace format and diff checks;
• focused tests for the changed optimizer, function, physical-expression, physical-plan, and Substrait crates;
• cargo clippy --all-targets --all-features -- -D warnings;
• DataFusion SQL logic tests, core/CLI tests, and the complete Substrait suite.

Actual completed evidence before submission:

• Verified ancestry from 0d1f2ebe2 and an exact 14-commit range.
• Range-diff maps the original 11 retained patches to the validated 54.0 stack and drops only the now-upstream volatile projection patch; the twelfth commit is the minimal 54.1 adaptation of Apache fix: Capture global ORDER BY requirement under ScalarSubqueryExec root apache/datafusion#23677. Commits 13–14 are the GreptimeTeam fix: preserve nulls in dictionary scalar regex #22 and fix: rewrite ordered timestamp widening predicates #23 ports (b1f616b1, d75a5582).
• Workspace cargo clippy --all-targets --all-features -- -D warnings.
• Scalar-subquery ordering rule-level and end-to-end physical-optimizer regressions, plus all-target/all-feature datafusion-physical-optimizer Clippy.
• fix: preserve nulls in dictionary scalar regex #22/fix: rewrite ordered timestamp widening predicates #23 validation: 8/8 named regression tests, simplify_expr.slt, full all-feature datafusion-expr-common/datafusion-optimizer/datafusion-physical-expr suites with 0 failures, cargo fmt --all -- --check, and full-workspace cargo clippy --all-targets --all-features -- -D warnings (0 warnings).

[discord9]

Closing per decision: GreptimeDB will consume the org thin-fork/2026-07-22-datafusion-54.1 branch directly, pinned at its current head d75a5582a7112e11bf2f36449f47fd60ac820a05 (already the rev in GreptimeDB's Cargo.toml). Future thin-fork advances will update that pinned rev directly; no PR review shell is needed.