Skip to content

refactor: Use cast preimages for cast predicate rewrites - #22906

Open
discord9 wants to merge 11 commits into
apache:mainfrom
discord9:experiment/cast-predicate-preimage
Open

refactor: Use cast preimages for cast predicate rewrites#22906
discord9 wants to merge 11 commits into
apache:mainfrom
discord9:experiment/cast-predicate-preimage

Conversation

@discord9

@discord9 discord9 commented Jun 11, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

The previous cast-unwrap path could only move the original comparison operator
from CAST(expr AS target_type) OP literal to expr OP casted_literal. That is
not correct for many-to-one casts such as timestamp precision narrowing, where
the source-domain preimage of one target value is a range rather than a
singleton.

For example, CAST(ts_ns AS Timestamp(ms)) > 1000ms must not become
ts_ns > 1_000_000_000ns; its exact source boundary is
ts_ns >= 1_001_000_000ns.

Timestamp precision widening has a related ordered-comparison case. A
non-aligned target literal has no singleton equality preimage, but it does have
an exact source-unit boundary for an ordered predicate. For example:

CAST(ts_ms AS Timestamp(ns)) >= 123_456_789ns

becomes:

ts_ms >= 124ms

This PR also makes exact cast rewrites closed-by-default: exact rewrites require
a supported value-preserving cast family. Many-to-one or source-domain-reducing
casts either use an explicit range/boundary preimage or remain unchanged.

The ordered timestamp-widening rewrite deliberately follows the existing
widening policy used by this work. At extreme source values where widening
overflows, ordinary CAST can error and TRY_CAST can return NULL, while the
rewritten source-unit comparison returns a Boolean. This is not a claim of
full-domain equivalence for those overflow cases; a guarded/error-aware
preimage representation is outside this PR's scope.

What changes are included in this PR?

  • Add a shared CastPredicatePreimage abstraction in
    datafusion-expr-common:
    • Exact(ScalarValue) for a same-operator source literal or boundary.
    • Range(Interval) for a half-open source-domain interval.
  • Share cast-preimage computation between logical and physical simplifiers.
  • Organize preimage computation into explicit range, special exact, generic
    exact, and ordered timestamp-widening paths.
  • Implement timestamp precision narrowing preimages using half-open buckets
    with truncation-toward-zero semantics, including negative timestamps.
  • Implement non-aligned ordered timestamp-widening boundaries using Euclidean
    floor/ceil arithmetic in i128:
    • >= L and < L use ceil(L / q).
    • > L and <= L use floor(L / q).
  • Support all timestamp precision-widening unit pairs with matching timezone
    metadata, including CAST, TRY_CAST, and literal-left comparisons.
  • Keep non-aligned equality, distinctness, and IN predicates unchanged.
  • Keep timestamp precision narrowing out of IN rewrites because its preimage
    is a range rather than a singleton.
  • Add conservative exact-cast family gates, including date, integer,
    signedness, decimal precision/scale, and canonical integer/string checks.
  • Replace the logical optimizer's cast-unwrap module with a cast-preimage module
    and update the physical simplifier to use the same helper.

Behavior changes compared to main

Expression Behavior after this PR Why
CAST(c1:Int32 AS Int64) < 10 c1 < Int32(10) Integer widening is value-preserving.
CAST(c2:Int64 AS Int32) = 5 kept Integer narrowing reduces the source domain.
CAST(c1:Int32 AS UInt32) = 5 kept Signed-to-unsigned is not full-domain safe.
CAST(c1:Int32 AS Utf8) = '123' c1 = Int32(123) The literal round-trips canonically.
CAST(c1:Int32 AS Utf8) = '0123' kept '0123' -> 123 -> '123' does not round-trip.
CAST(c1:Int32 AS Utf8) < '123' kept String ordering is not integer ordering.
CAST(c1:Int32 AS Decimal(12,2)) = 123.00 c1 = Int32(123) The target decimal represents the full source domain.
CAST(c1:Int32 AS Decimal(10,2)) = 123.00 kept The target has insufficient integer digits.
CAST(c3:Decimal(10,2) AS Decimal(18,4)) = 123.0000 exact rewrite Precision/scale widening is value-preserving.
CAST(c3:Decimal(18,2) AS Decimal(18,1)) = 123.0 kept Scale narrowing is many-to-one.
CAST(ts_ns AS timestamp(ms)) = 1000ms ts_ns >= 1_000_000_000ns AND ts_ns < 1_001_000_000ns Equality preimage is a timestamp bucket.
CAST(ts_ns AS timestamp(ms)) > 1000ms ts_ns >= 1_001_000_000ns Uses the bucket's upper boundary.
CAST(ts_ns AS timestamp(ms)) <= 0ms ts_ns < 1_000_000ns The zero bucket follows truncation toward zero.
CAST(ts_ns AS timestamp(ms)) = -1ms ts_ns >= -1_999_999ns AND ts_ns < -999_999ns Negative buckets follow truncation toward zero.
CAST(ts_ns AS timestamp(ms)) != 1000ms range-complement rewrite Complements the timestamp bucket.
CAST(ts_ns AS timestamp(ms)) IN (1000ms) kept IN currently supports only singleton exact preimages.
CAST(ts_ms AS timestamp(ns)) = 123_000_000ns ts_ms = 123ms The aligned widening literal round-trips.
CAST(ts_ms AS timestamp(ns)) = 123_456_789ns kept A non-aligned literal has no equality preimage.
CAST(ts_ms AS timestamp(ns)) >= 123_456_789ns ts_ms >= 124ms Ordered widening uses the exact Euclidean source boundary under the documented overflow policy.
123_456_789ns < TRY_CAST(ts_ms AS timestamp(ns)) ts_ms > 123ms Literal-left operators are swapped before computing the same boundary.
CAST(Date64_col AS Date32) = ... kept Date precision narrowing is not exact-safe.
CAST(Date32_col AS Date64) = aligned_midnight exact rewrite Date widening is injective when the literal is on a day boundary.

Are these changes tested?

Yes. Tests cover:

  • exact and range preimages and conservative cast-family gating,
  • timestamp narrowing for positive, zero, and negative values,
  • all timestamp widening unit pairs,
  • aligned and non-aligned widening literals,
  • all four ordered operators with positive and negative values,
  • CAST, TRY_CAST, and literal-left forms,
  • timezone/type/NULL rejection and i64 boundary literals,
  • equality, distinctness, and IN remaining unchanged where required,
  • logical, physical, and sqllogictest plan output.

Validated locally with:

cargo test -p datafusion-expr-common --lib cast_predicate
cargo test -p datafusion-optimizer --lib cast_preimage
cargo test -p datafusion-physical-expr --lib unwrap_cast
cargo test -p datafusion-sqllogictest --test sqllogictests -- simplify_expr
cargo fmt --all -- --check
cargo clippy --all-targets --all-features -- -D warnings

Are there any user-facing changes?

There are no public API changes. Optimized plans may now use exact source-domain
ranges or boundaries for cast predicates, and previously unsafe exact rewrites
may remain unchanged. Ordered timestamp-widening comparisons also follow the
explicit overflow policy described above.

@github-actions github-actions Bot added logical-expr Logical plan and expressions physical-expr Changes to the physical-expr crates optimizer Optimizer rules labels Jun 11, 2026
@discord9 discord9 changed the title Use cast preimages for cast predicate rewrites refactor: Use cast preimages for cast predicate rewrites Jun 11, 2026
@github-actions github-actions Bot added the sqllogictest SQL Logic Tests (.slt) label Jun 11, 2026
@2010YOUY01

Copy link
Copy Markdown
Contributor

This PR looks like a very nice solution for the cast pattern. I'm comfortable proceeding with it, but please forgive me for briefly advocating an alternative approach (that I'm to happy to help reviewing or implementing):

I believe the fundamental goal here is to enable pruning through nested expressions, and the propagation based approach could be a better long term solution.

My concern with the preimage approach is that it requires introducing and maintaining an ever-growing set of reverse-transformation rules. Even with additional rules, there will likely still be cases that cannot be handled. If this becomes a supported pattern, I worry that the long-term maintenance burden could be significant.

In contrast, the propagation approach seems both more general and easier to reason about. The key intuition is that it follows a forward-evaluation model, similar to normal expression evaluation, whereas the preimage approach attempts to reverse complex expressions back into a simpler form. In many cases, the latter is inherently more difficult and may require expression-specific logic.

@alamb

alamb commented Jun 22, 2026

Copy link
Copy Markdown
Contributor

I think this idea shows promise -- I will review it more carefully shortly

@discord9
discord9 force-pushed the experiment/cast-predicate-preimage branch from a292aab to 17d45ee Compare June 25, 2026 08:26
@discord9

discord9 commented Jul 1, 2026

Copy link
Copy Markdown
Contributor Author

Hi @alamb, quick update: I rebased this PR onto latest main and all CI checks are green now. No rush, but when you get a chance I'd appreciate your review.

@alamb

alamb commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

Thank you -- I will try and review it shorlty.

@codecov-commenter

codecov-commenter commented Jul 22, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 94.87566% with 68 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.77%. Comparing base (68d5874) to head (9adf02a).
⚠️ Report is 7 commits behind head on main.

Files with missing lines Patch % Lines
datafusion/expr-common/src/casts.rs 94.46% 23 Missing and 14 partials ⚠️
...ptimizer/src/simplify_expressions/cast_preimage.rs 95.94% 11 Missing and 4 partials ⚠️
...fusion/physical-expr/src/simplifier/unwrap_cast.rs 95.57% 9 Missing and 3 partials ⚠️
...imizer/src/simplify_expressions/expr_simplifier.rs 81.81% 0 Missing and 2 partials ⚠️
datafusion/physical-expr/src/simplifier/mod.rs 66.66% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #22906      +/-   ##
==========================================
+ Coverage   80.75%   80.77%   +0.02%     
==========================================
  Files        1096     1096              
  Lines      373282   374438    +1156     
  Branches   373282   374438    +1156     
==========================================
+ Hits       301440   302470    +1030     
- Misses      53869    53940      +71     
- Partials    17973    18028      +55     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

discord9 added 9 commits July 29, 2026 16:32
Signed-off-by: discord9 <discord9@163.com>
Signed-off-by: discord9 <discord9@163.com>
Signed-off-by: discord9 <discord9@163.com>
Signed-off-by: discord9 <discord9@163.com>
Signed-off-by: discord9 <discord9@163.com>
@discord9
discord9 force-pushed the experiment/cast-predicate-preimage branch from 574c724 to afe8c38 Compare July 29, 2026 08:54
@discord9

Copy link
Copy Markdown
Contributor Author

One scope question before review: the latest push includes the ordered
timestamp-widening case from
GreptimeTeam/datafusion#23,
but I am unsure whether it is better to keep that case in this PR or split it
into a follow-up.

The motivating shape appears after mixed timestamp coercion, for example:

CAST(ts_ms AS Timestamp(ns)) >= TimestampNanosecond(123456789)

The non-aligned literal has no singleton equality preimage, so equality and
IN remain unchanged. Ordered comparisons do have an exact source-unit bound:

ts_ms >= TimestampMillisecond(124)

More generally, for widening ratio q and target literal L:

  • >= L and < L use ceil(L / q);
  • > L and <= L use floor(L / q).

The implementation uses Euclidean i128 arithmetic, supports all timestamp
unit-widening pairs with matching timezone metadata, and covers positive and
negative literals, CAST / TRY_CAST, literal-left comparisons, and boundary
values. Keeping it here is attractive because both logical and physical paths
can reuse the shared cast-preimage abstraction introduced by this PR rather
than implementing separate rewrite logic.

There is an important policy caveat: for extreme source values where widening
overflows, ordinary CAST can error and TRY_CAST can return NULL, while the
rewritten source-unit comparison returns a Boolean. The fork PR deliberately
accepts and documents that existing widening policy; a fully equivalent
upstream solution would require a richer guarded/error-aware preimage model.

So I see two reasonable choices:

  1. Keep the ordered widening support here because it is a natural use of the
    new preimage abstraction and fixes the motivating predicate-pushdown case.
  2. Keep this already-large PR focused on the abstraction/narrowing work and
    split ordered widening (and its explicit overflow policy) into a follow-up.

I am happy to keep the latest commits or split them back out, depending on what
maintainers would find easier to review and merge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

logical-expr Logical plan and expressions optimizer Optimizer rules physical-expr Changes to the physical-expr crates sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Unsafe comparison cast rewriting in ExprSimplifier silently produces wrong query results

4 participants