Skip to content

perf(encode): close the L6-L9 Row/lazy dictionary speed gap #434

Description

@polaz

Problem

After the small-frame FSE-table-selection fix (PR #433), Fast/dfast dictionary frames are at or below libzstd parity on both arches. The remaining dictionary-path gap is at the Row / lazy backends (levels ~6-9), which stay 2-3x slower than libzstd and also lose a little ratio.

Authoritative compare numbers (ours vs libzstd compress_usingCDict, same host, dict_matrix example, raw-content 8 KiB dict, dict-id flag off):

fixture lvl i9 (x86) sp M1 (aarch64) sp ratio
logs-1k 6 2.75x 2.10x 1.44x (26 vs 18 B)
logs-1k 9 2.02x 2.00x 1.00x
logs-4k 9 2.05x 1.94x 1.00x
logs-16k 9 1.80x 1.86x 1.00x

(sp = ours_us / c_us; >1 = slower. ratio = our_bytes / c_bytes.)

Scope

  • Profile the L6 (Row/greedy-lazy) and L9 (HashChain/lazy2) dictionary path on both arches (x86 BMI2 on the i9, NEON on the M1 — a same-hardware ours-vs-libzstd delta is authoritative per arch).
  • L6 also shows a ratio gap (we emit 26 vs libzstd's 18 bytes on logs-1k): the lazy matcher finds fewer/shorter dictionary matches on the dict-primed chains. Ratio is the first priority.
  • Likely targets: the Row tag-mask SIMD coverage, dict-primed chain traversal depth, and the lazy carry-forward decision on dict matches.

Acceptance

  • L6/L9 dictionary throughput within ~1.2x of libzstd on the bench fixtures (both arches), ratio >= libzstd.
  • No regression on the Fast/dfast/BT dictionary paths or non-dict levels.

Follow-up to PR #433 (which fixed the 5.5x copy-mode Fast dict ratio bug, the resident-dict retained-budget over-window bug, and the small-frame FSE-selection hotspot).

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1-highHigh priority — core functionalityenhancementNew feature or requestperformancePerformance optimization

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions