fix: fold -0.0 into +0.0 in IN-list filters and the distinct accumulator - #4
Open
fornwall wants to merge 1 commit into
Open
fix: fold -0.0 into +0.0 in IN-list filters and the distinct accumulator#4fornwall wants to merge 1 commit into
fornwall wants to merge 1 commit into
Conversation
`=` already normalizes float zeros before comparing (IEEE 754 and PostgreSQL both treat `-0.0` and `+0.0` as equal), and so do GROUP BY and joins, but two paths still compared raw bits: the IN-list static filters (hash set / bitmap) and dynamic list path, and the distinct-values accumulator used by e.g. COUNT(DISTINCT) alongside other aggregates. This made `0.0 IN (-0.0)` false while `0.0 = -0.0` is true, and made `SELECT COUNT(DISTINCT x), COUNT(x) FROM (VALUES (0.0), (-0.0)) t(x)` return 2 where the GROUP BY rewrite of the same query returns 1. Normalize with the existing `normalize_float_zero` helpers at those sites. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WS4N9hte6f1W5r6EuDzobY
This was referenced Jul 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
No tracking issue — this is a follow-up to apache#22835, which introduced
-0.0 → +0.0normalization for float equality and grouping. Happy to file an issue first if preferred.Rationale for this change
apache#22835 normalized float zeros at the sites where DataFusion hands float data to Arrow's totalOrder-based kernels (
=, GROUP BY, joins, array set ops), so-0.0and+0.0compare and group as equal, matching IEEE 754 equality and PostgreSQL. A few paths were missed and still compare raw bits, so the engine currently disagrees with itself:The second inconsistency is the same class as apache#16254 (COUNT(DISTINCT) giving different answers for the same data depending on which execution path runs).
What changes are included in this PR?
Apply the existing
normalize_float_zero/normalize_float_zero_scalarhelpers from apache#22835 at the remaining sites:OrderedFloat32/OrderedFloat64fold-0.0on construction (covering both set build and probe), and theFloat16BitmapFilterfolds it inindex().eqkernel / comparator.GenericDistinctBuffer::update_batch: normalize input arrays before inserting into the bit-basedHashableset (merge_batchdelegates toupdate_batch).No new helpers, no semantic policy change — only making the remaining paths agree with the semantics apache#22835 already established.
Are these changes tested?
predicates.slt:INover±0.0on both the constant-list (static filter) and non-constant-list paths, andCOUNT(DISTINCT)with mixed aggregates so the distinct accumulator runs.Float16bitmap filter unit test now probes+0.0against a-0.0haystack entry.Are there any user-facing changes?
The queries above now return consistent results (
0.0 IN (-0.0)is true; mixed-aggregateCOUNT(DISTINCT)counts±0.0as one value). No API changes.🤖 Generated with Claude Code
https://claude.ai/code/session_01WS4N9hte6f1W5r6EuDzobY