Problem
At level 1 (fast) the encoder wins on large throughput inputs but is slow on
small ones. compare_ffi, level_1_fast:
small-4k-log-lines: rust 30.4 us vs C 7.2 us = 4.2x slower (ratio fine: 156 < 157)high-entropy-1m: rust wins 1.58x; low-entropy-1m: wins 1.19x;
large-log-stream: wins 1.18x
So the fast match loop itself is fine; the small-input gap is per-frame
fixed cost.
Profile (perf, encode_loop_dict 1 ... logs4096, self-time)
incompressible::update_sample_metrics 21.3% — the raw-fast-path
pre-scan reads up to a 4 KiB sample on every block, even for clearly
compressible data__memset_avx2 13.8% — per-frame fast hash-table zeroing
(fast_hash_log = 14 -> 64 KiB cleared for a 4 KiB input)- entropy table build (Huffman + FSE) ~25% + sorts ~6.5%
run_fast_kernel_block (the actual match loop) only 3.9%
Levers (all SPEED, ratio must not regress)
- Incompressible pre-scan early-exit. The verdict is
distinct >= 200 && max_freq <= sample/24 && repeats <= quads/64. The two
upper-bound guards are fixed before scanning and both tracked quantities
only grow, so the scan can stop the instant either guard is passed — the
exact same verdict, but compressible blocks bail in the first few hundred
bytes instead of scanning the whole 4 KiB sample.
- Right-size the fast hash table for small inputs so the per-frame
memset scales with the source (the reference's ZSTD_adjustCParams
shape), instead of always clearing the full 1 << 14 table.
- Cheaper entropy path for tiny outputs (the table build dominates a
~156-byte block).
Lever 1 is exact (verdict-preserving) and lands first; 2 and 3 are
follow-ups.
Problem
At level 1 (fast) the encoder wins on large throughput inputs but is slow on
small ones.
compare_ffi,level_1_fast:small-4k-log-lines: rust 30.4 us vs C 7.2 us = 4.2x slower (ratio fine: 156 < 157)high-entropy-1m: rust wins 1.58x;low-entropy-1m: wins 1.19x;large-log-stream: wins 1.18xSo the fast match loop itself is fine; the small-input gap is per-frame
fixed cost.
Profile (perf,
encode_loop_dict 1 ... logs4096, self-time)incompressible::update_sample_metrics21.3% — the raw-fast-pathpre-scan reads up to a 4 KiB sample on every block, even for clearly
compressible data
__memset_avx213.8% — per-frame fast hash-table zeroing(
fast_hash_log = 14-> 64 KiB cleared for a 4 KiB input)run_fast_kernel_block(the actual match loop) only 3.9%Levers (all SPEED, ratio must not regress)
distinct >= 200 && max_freq <= sample/24 && repeats <= quads/64. The twoupper-bound guards are fixed before scanning and both tracked quantities
only grow, so the scan can stop the instant either guard is passed — the
exact same verdict, but compressible blocks bail in the first few hundred
bytes instead of scanning the whole 4 KiB sample.
memset scales with the source (the reference's
ZSTD_adjustCParamsshape), instead of always clearing the full
1 << 14table.~156-byte block).
Lever 1 is exact (verdict-preserving) and lands first; 2 and 3 are
follow-ups.