Feat: add dictionaries as a supported group column type - #23187
Open
Rich-T-kid wants to merge 6 commits into
Open
Feat: add dictionaries as a supported group column type#23187Rich-T-kid wants to merge 6 commits into
Rich-T-kid wants to merge 6 commits into
Conversation
Rich-T-kid
force-pushed
the
rich-T-kid/dictionary-groupValuesColumn-impl
branch
2 times, most recently
from
July 1, 2026 04:25
3f7ff57 to
e6b6dce
Compare
Contributor
Author
|
@kumarUjjawal could you run the dictionary benchmarks on this PR? Thx |
Rich-T-kid
commented
Jul 1, 2026
| } | ||
| } | ||
| DataType::Dictionary(key_dt, value_dt) => { | ||
| let new_field = Field::new("", *value_dt.clone(), true); |
Contributor
Author
There was a problem hiding this comment.
Since this field is never read again it may be fine to ignore the name field.
should be weary of similar issues to #21765 (comment)
Contributor
There was a problem hiding this comment.
Kind of annoying that make_group_column takes a field instead of a DataType. Maybe we can change that in a follow up PR?
Rich-T-kid
commented
Jul 1, 2026
Rich-T-kid
commented
Jul 1, 2026
Contributor
Author
|
@kumarUjjawal wanted to bump this 😄 |
Contributor
|
run benchmark dictionary_group_values |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing rich-T-kid/dictionary-groupValuesColumn-impl (eb41915) to 01bf68c (merge-base) diff using: dictionary_group_values File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usagedictionary_group_values — base (merge-base)
dictionary_group_values — branch
File an issue against this benchmark runner |
Rich-T-kid
force-pushed
the
rich-T-kid/dictionary-groupValuesColumn-impl
branch
2 times, most recently
from
July 1, 2026 15:32
770abfe to
243a557
Compare
Contributor
Author
|
@codex review |
Contributor
@Rich-T-kid Thank you! I have been sick so I won't be available for review. I will probably get back next week. |
Contributor
Author
@kumarUjjawal Sorry to hear that. I hope you feel better! no rush on the review! |
Rich-T-kid
commented
Jul 2, 2026
Rich-T-kid
force-pushed
the
rich-T-kid/dictionary-groupValuesColumn-impl
branch
from
July 2, 2026 19:18
4ee52da to
7af7080
Compare
Rich-T-kid
commented
Jul 4, 2026
Rich-T-kid
force-pushed
the
rich-T-kid/dictionary-groupValuesColumn-impl
branch
from
July 5, 2026 03:59
152c1f0 to
f3387c5
Compare
Contributor
Author
Latest PR speeds up the low cardinality case
|
Contributor
Author
|
@geoffreyclaude could you run the benchmarks command again when you get a chance. Thanks 🚀 |
Contributor
|
run benchmark dictionary_group_values |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing rich-T-kid/dictionary-groupValuesColumn-impl (3d1e1c9) to 01bf68c (merge-base) diff using: dictionary_group_values File an issue against this benchmark runner |
Rich-T-kid
commented
Jul 6, 2026
Rich-T-kid
left a comment
Contributor
Author
There was a problem hiding this comment.
Revision #4 Last revision before making this open for review
| cached_combined: Option<ArrayRef>, | ||
| /// Per-call group equality cache for the low-cardinality path. `Mutex` | ||
| /// because `vectorized_equal_to` takes `&self`; always uncontended. | ||
| group_eq_cache: Mutex<Vec<Option<bool>>>, |
Contributor
Author
There was a problem hiding this comment.
This is a work around needed because vectorized_equal_to takes in a &self as opposed to &mut self. This means the alternative is needing to allocate target_batch_size arrays for each intern() call instead of re-using the vector.
Originally planned to use RefCell but its not Send + Sync
Contributor
Author
There was a problem hiding this comment.
this shouldn't incur any overhead since there no contention but I'm happy to explore other ideas.
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usagedictionary_group_values — base (merge-base)
dictionary_group_values — branch
File an issue against this benchmark runner |
Contributor
Author
|
🤔 benchmarks show good improvement in every case but it should be much larger. Currently every |
zhuqi-lucas
reviewed
Jul 27, 2026
zhuqi-lucas
reviewed
Jul 27, 2026
Rich-T-kid
force-pushed
the
rich-T-kid/dictionary-groupValuesColumn-impl
branch
from
July 27, 2026 14:43
38d656f to
fb60ab3
Compare
Contributor
Author
|
thank you for the review @zhuqi-lucas, the two latest PR's should address your comments + test. @kumarUjjawal @zhuqi-lucas can we run the benchmarks one more time to make sure theres no meaningful regressions |
Contributor
|
run benchmark dictionary_group_values |
|
Benchmark for this request failed. Last 20 lines of output: Click to expandFile an issue against this benchmark runner |
Contributor
|
run benchmark dictionary_group_values |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing rich-T-kid/dictionary-groupValuesColumn-impl (e994180) to bb75d92 (merge-base) diff using: dictionary_group_values File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usagedictionary_group_values — base (merge-base)
dictionary_group_values — branch
File an issue against this benchmark runner |
Contributor
Author
|
nice no regressions 🚀 |
| let old_inner_len = self.inner.len(); | ||
| let all_inner_values = self.inner.take_n(old_inner_len); | ||
|
|
||
| let emitted = |
Contributor
There was a problem hiding this comment.
Draining all inner values may be required by the trait, but returning all of them in the emitted dictionary is not. Once all_inner_values is available, the values referenced by the emitted groups can be compacted and their keys remapped separately from the remaining groups.
The bytes example copies the remaining data, but its output contains only the requested prefix. Here, every partial batch still retains and may serialize values needed only by later batches, so the quadratic output cost remains.
7 tasks
add schema support for dictionarys introduce high level GroupValuesColumn test introduce groupColumn trait test git issues introduce edge case/ regression section inital impl working implementation of dictionary for groupValuesCOlumns benchmarks show perf boost over groupvaluerows, TODO:dedupe items before inner append fix clippy errors & inline final builder add cache for arc ptr trim down test trim test LOC again trim PR revision 3 speed up low cardinlaity case working version introduce inter-batch caching break complex types into seperate parts fixed breaking test, re-allocate hashtable on each intern() call re-introduce cache remove mutex add cache to concat pointers to avoid un-needed allocations remove ptr caches and concat call reduce LOC revised PR comments add regression test to align with GroupValueRows & re-order overflow check add test to assert de-duplicated output dictionary tmp low card speed up optimize low-card case wip re-use allocations across calls add test final clean up
Rich-T-kid
force-pushed
the
rich-T-kid/dictionary-groupValuesColumn-impl
branch
from
July 31, 2026 04:16
e994180 to
cc5923e
Compare
Rich-T-kid
force-pushed
the
rich-T-kid/dictionary-groupValuesColumn-impl
branch
from
July 31, 2026 04:19
cc5923e to
c98c47f
Compare
Contributor
Author
|
@kumarUjjawal commit c98c47f:
is there anything else? |
Contributor
|
run benchmark dictionary_group_values |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing rich-T-kid/dictionary-groupValuesColumn-impl (c98c47f) to 30ae8bf (merge-base) diff Run configurationrun benchmark dictionary_group_valuesResults will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing rich-T-kid/dictionary-groupValuesColumn-impl (c98c47f) to 30ae8bf (merge-base) diff Run configurationrun benchmark dictionary_group_valuesCPU Details (lscpu)Details
Resource Usagedictionary_group_values — base (merge-base)
dictionary_group_values — branch
File an issue against this benchmark runner |
zhuqi-lucas
approved these changes
Jul 31, 2026
| if all_inner_values.is_null(old) { | ||
| None | ||
| } else { | ||
| Some(K::Native::usize_as(emit_old_to_new[old])) |
Contributor
There was a problem hiding this comment.
Null participates in emitted slot ordering. At key capacity, the final value gets index 128 for Int8 or 256 for UInt8. Int8 panics; UInt8 wraps and returns the wrong value. Move null last or exclude it. Add repeated boundary-emission tests.
| } | ||
|
|
||
| fn take_n(&mut self, n: usize) -> ArrayRef { | ||
| let old_inner_len = self.inner.len(); |
Contributor
There was a problem hiding this comment.
Every partial emit still drains, hashes, and rebuilds all surviving values, retaining the O(G² / batch_size) cost and high peak memory. Also, Arrow take retains full backing storage for Utf8View, BinaryView, and nested dictionaries.
| // Keys are raw slot indices. The null slot is excluded from key count | ||
| // only when it occupies the last position — any non-null slot above it | ||
| // still emits that slot's raw index as a key. | ||
| let inner_len = self.inner.len(); |
Contributor
There was a problem hiding this comment.
Null first plus the maximum non-null values fails, while the same values with null last succeeds.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
This PR introduces a specialized
GroupColumnimplementation for dictionary-typed columns insideGroupValuesColumn, allowing dictionary columns to participate in the columnar, vectorized aggregation path instead of the row-based fallback.The Implementation is only about 175+ lines of code. the remaining LOC is adding extensive test at the
GroupColumntrait level as well as testing theGroupValuesColumnGroupValues trait and how it inter-opts with multi-dictionary group by's.What changes are included in this PR?
DictionaryGroupValueBuilderstruct implementing theGroupColumntrait forDictionary-typed group-by columns, supporting a configurable subset of value typesGroupValuesColumn::try_new(thematches!block) to acceptDictionary(_, value_type)wherevalue_typeis already supported.emitAre these changes tested?
yes. a majority of this PR is test
Are there any user-facing changes?
no. this is a pure perf boost for users.