refactor: purge reference-impl terminology + isolate C bindings in ffi-bench

[polaz]

Summary

Two related cleanups across the whole repo:

1. Terminology — removed the legacy reference-implementation term from every identifier, file name, comment, README, and Cargo metadata. Names now describe what a value is / what a function does, never where it came from. Reference attribution (file:line citations) stays in prose comments only.

2. C bindings isolated — the library crate (structured-zstd) now links zero C bindings. Every libzstd cross-check (benches, FFI parity tests, profiling examples) lives in a new non-published ffi-bench crate that depends on both structured-zstd and zstd (the C bindings).

C-binding test migration

All C-binding unit tests moved out of the library into ffi-bench/tests/:

• frame compressor FFI roundtrips (+ fuzz-artifact interop replay)
• streaming encoder FFI roundtrips
• huff0 weight-description / 4-stream / level-22 literal-section conformance
• match-generator level-22 sequence-stream conformance
• dictionary roundtrip through the C decoder
• corpus roundtrip through the C decoder

White-box cases keep their pure-Rust capture in the library behind a bench_internals testing facade and feed only the result to the C side in ffi-bench. The our-decoder corpus roundtrips stay in the library. The zstd dev-dependency is removed from the library crate.

Verification

• cargo tree -p structured-zstd --all-features -i zstd-sys → not in the dependency graph
• no zstd_sys / zstd:: references remain in zstd/src/
• structured-zstd: 812 tests pass (no C dev-dependency)
• ffi-bench: 53 tests pass
• whole workspace builds; cargo fmt --check clean

Summary by CodeRabbit

• Chores

  • Added an ffi-bench workspace crate to run C FFI benchmarking, conformance checks, and interoperability validation.
  • Reorganized CI and benchmark/test wiring to improve Rust-vs-C parity coverage and lint depth.
  • Updated documentation terminology for consistent “upstream” references.

• Bug Fixes

  • Improved compression framing behavior: the encoder now falls back to Raw blocks when the “compressed” output would not shrink, for both owned and borrowed paths.

• Build

  • Adjusted C API crate outputs to emit only cdylib and staticlib.

[coderabbitai]

📝 Walkthrough

Walkthrough

This PR adds an ffi-bench workspace crate, moves FFI-oriented tests and targets into it, exposes bench-internals helpers, updates several upstream-parity internals and references, and rewrites donor terminology to upstream zstd wording across code, tests, docs, examples, benchmarks, and workflow comments.

Changes

FFI bench extraction and upstream parity

Layer / File(s)	Summary
Workspace, CI, and benchmark wiring Cargo.toml, zstd/Cargo.toml, ffi-bench/Cargo.toml, .github/workflows/ci.yml, .github/scripts/run-benchmarks.sh, zstd/examples/*, zstd/benches/*, c-api/Cargo.toml, .github/bench-dashboard/index.html	Adds the ffi-bench crate to the workspace, disables auto target discovery in zstd, declares bench/test/example targets from the new crate, and updates CI/benchmark commands and comments to use the new package split.
FFI conformance and replay tests ffi-bench/tests/*, zstd/src/encoding/frame_compressor.rs, zstd/src/encoding/streaming_encoder.rs, zstd/src/tests/encode_corpus.rs, zstd/tests/block_splitter_parity.rs, zstd/tests/corpus_level22_gap.rs, zstd/tests/cross_validation.rs	Adds dedicated C-interop tests for dictionaries, corpus roundtrips, fuzz replay, Huffman payloads, streaming encoding, block splitting, and level-22 sequences; removes or redirects equivalent in-crate FFI tests; updates some fixture resolution and reference targets.
Bench-internals helper surface zstd/src/lib.rs, zstd/src/dictionary/mod.rs, zstd/src/huff0/huff0_encoder.rs, zstd/src/encoding/match_generator.rs, zstd/src/encoding/match_table/storage.rs, ffi-bench/src/lib.rs	Adds internal helper functions for dictionary roundtrips, Huffman payload extraction, frame inspection, and level-22 sequence collection, plus renames one internal helper to upstream-focused naming.
Parity-sensitive encoding and decoding paths zstd/src/encoding/*, zstd/src/decoding/*, zstd/src/fse/*, zstd/src/huff0/*, zstd/src/dictionary/*	Adjusts Fast kernel search strength, dfast rep-extension and window bookkeeping, iterative LDM table rebasing, Huffman limited-weight helper usage, FSE normalization/helper naming, and multiple decoding/encoding boundary checks to align with upstream zstd behavior.
Documentation, comments, and naming cleanup README.md, docs/SKIPPABLE_MAGIC_ALLOCATIONS.md, zstd/src/**/*, zstd/tests/*, zstd/benches/*, zstd/examples/*	Rewrites donor references to upstream zstd or inline-path wording across documentation comments, examples, tests, benchmarks, error text, and module docs without changing most runtime logic in those files.

Estimated code review effort

🎯 5 (Critical) | ⏱️ ~120 minutes

Possibly related PRs

• structured-world/structured-zstd#38: Both PRs touch the benchmark harness and run-benchmarks.sh flow around compare_ffi and compare_ffi_memory.
• structured-world/structured-zstd#44: The new dictionary roundtrip fixture and dictionary-backed FFI test path depend on the dictionary attachment and validation support added there.
• structured-world/structured-zstd#94: Both PRs touch the one-shot framing and hinting path in zstd/src/encoding/frame_compressor.rs and the corresponding C-compatibility checks.

Poem

🐇 I packed up benches in a tidy new warren,
With upstream names where donors were worn.
I tested the frames, the Huffman, the stream,
And chased level 22 through a bytey dream.
Now carrots of parity grow crisp and bright.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch refactor/upstream-terminology

[codecov]

Codecov Report

❌ Patch coverage is 97.43590% with 3 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
zstd/src/huff0/huff0_encoder.rs	33.33%	2 Missing ⚠️
zstd/src/encoding/match_generator.rs	98.07%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[coderabbitai]

Actionable comments posted: 5

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (2)

zstd/src/decoding/errors.rs (1)

1013-1050: ⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Use backend-agnostic wording for OutputBufferOverflow.

zstd/src/decoding/flat_buf.rs::exec_sequence_inline also raises this variant, so "fixed-size buffer" / UserSliceBackend is too narrow and will misreport valid capacity failures. Please keep the enum doc and Display text aligned with the actual callers.

💡 Suggested text

- Inline sequence would write past fixed-size buffer: tail={tail}, requested={requested}, capacity={capacity}
+ Inline sequence would write past the output buffer: tail={tail}, requested={requested}, capacity={capacity}

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@zstd/src/decoding/errors.rs` around lines 1013 - 1050, The error message for
the OutputBufferOverflow variant in the Display implementation for
ExecuteSequencesError uses backend-specific language ("fixed-size buffer" and
references to UserSliceBackend) that does not accurately reflect all callers of
this error. Update the write! macro invocation for the OutputBufferOverflow
match arm to use backend-agnostic terminology that describes the overflow
condition without assuming a specific buffer type, and ensure the enum
documentation comment for OutputBufferOverflow is also updated to match this
broader scope.

zstd/src/encoding/match_generator.rs (1)

4078-4093: ⚠️ Potential issue | 🟡 Minor

Add feature = "std" to the cfg gate for x86 feature detection code.

The std::is_x86_feature_detected! macro requires std, but the test block at line 4078 only gates on target architecture. Other modules in the codebase (fastpath/mod.rs, sequence_section_decoder.rs) consistently gate the same macro with #[cfg(all(feature = "std", ...))]. Update this test to follow the established pattern and ensure it compiles in no_std builds.

Suggested fix

-    #[cfg(any(target_arch = "x86", target_arch = "x86_64"))]
+    #[cfg(all(feature = "std", any(target_arch = "x86", target_arch = "x86_64")))]
     {
         if std::is_x86_feature_detected!("sse4.2") {

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@zstd/src/encoding/match_generator.rs` around lines 4078 - 4093, The cfg gate
for the x86 feature detection block starting with `#[cfg(any(target_arch =
"x86", target_arch = "x86_64"))]` is missing the required `feature = "std"`
condition. The `std::is_x86_feature_detected!` macro depends on the std feature,
so update the cfg attribute to include `feature = "std"` alongside the existing
target architecture check using an all() wrapper to match the established
pattern used in other modules like fastpath/mod.rs and
sequence_section_decoder.rs. This ensures the code compiles correctly in no_std
builds.

Source: Pipeline failures

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@ffi-bench/Cargo.toml`:
- Line 11: The edition field in the Cargo.toml is set to an invalid value of
"2024". Rust only supports editions 2015, 2018, and 2021. Change the edition
field from "2024" to "2021" to use the most recent stable edition.
- Around line 27-38: The features section in the Cargo.toml file is missing the
required `std` feature that coding guidelines mandate. Add a new `std` feature
line to the [features] section and include it in the default features array
alongside the existing features like hash, dict_builder, bench_internals, and
copy_shape_stats. The `std` feature should also forward to the corresponding
feature in the structured-zstd dependency, following the same pattern as the
other features already defined.

In `@ffi-bench/tests/encode_corpus_ffi.rs`:
- Around line 18-24: The corpus_files function collects all directory entries
from both CORPUS_DIR and LOCAL_CORPUS_DIR without filtering by file type, which
causes a panic when the test later attempts to read non-file entries like
subdirectories or symlinks with fs::read(). Filter the collected entries to only
include regular files by checking the metadata of each directory entry before
adding its path to the files vector. This filtering should be applied to both
the initial read_dir(CORPUS_DIR) iteration and the extension from
read_dir(LOCAL_CORPUS_DIR) to ensure only actual files are included.

In `@ffi-bench/tests/huff0_ffi.rs`:
- Around line 197-202: The literal compressed size field extraction in the
lhl_code=2 case is missing a bit mask. In the fourth tuple element where
lit_c_size is calculated as (header >> 18) as usize, apply a 14-bit mask
(0x3fff) to ensure only bits 18-31 are captured, similar to the pattern used in
the lhl_code=1 case above it. Change (header >> 18) as usize to ((header >> 18)
& 0x3fff) as usize to prevent upper bits from being included in the result.

In `@zstd/src/encoding/blocks/compressed.rs`:
- Around line 531-544: The all-identical literal shortcut is currently emitting
RLE for every non-empty run, including 1-5 byte inputs where upstream zstd would
emit raw format instead. This breaks byte-for-byte parity with upstream zstd.
Add a minimum size gate before the RLE emission logic to ensure small
all-identical runs (below the threshold where RLE becomes beneficial) fall
through to raw encoding instead. Apply the same size check logic to the
estimate_literals_section_bytes function to keep the byte estimation aligned
with actual encoding behavior. Reference the comment's explanation about when
RLE equals raw at len==1 and becomes smaller only for len>=2 to determine the
appropriate gate condition.

---

Outside diff comments:
In `@zstd/src/decoding/errors.rs`:
- Around line 1013-1050: The error message for the OutputBufferOverflow variant
in the Display implementation for ExecuteSequencesError uses backend-specific
language ("fixed-size buffer" and references to UserSliceBackend) that does not
accurately reflect all callers of this error. Update the write! macro invocation
for the OutputBufferOverflow match arm to use backend-agnostic terminology that
describes the overflow condition without assuming a specific buffer type, and
ensure the enum documentation comment for OutputBufferOverflow is also updated
to match this broader scope.

In `@zstd/src/encoding/match_generator.rs`:
- Around line 4078-4093: The cfg gate for the x86 feature detection block
starting with `#[cfg(any(target_arch = "x86", target_arch = "x86_64"))]` is
missing the required `feature = "std"` condition. The
`std::is_x86_feature_detected!` macro depends on the std feature, so update the
cfg attribute to include `feature = "std"` alongside the existing target
architecture check using an all() wrapper to match the established pattern used
in other modules like fastpath/mod.rs and sequence_section_decoder.rs. This
ensures the code compiles correctly in no_std builds.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: f74e9d73-54ce-48e0-97f9-e75cc108f03d

📥 Commits

Reviewing files that changed from the base of the PR and between e0ca9c9 and a98714b.

📒 Files selected for processing (97)

• .github/bench-dashboard/index.html
• .github/workflows/ci.yml
• Cargo.toml
• README.md
• docs/SKIPPABLE_MAGIC_ALLOCATIONS.md
• ffi-bench/Cargo.toml
• ffi-bench/src/lib.rs
• ffi-bench/tests/dictionary_ffi.rs
• ffi-bench/tests/encode_corpus_ffi.rs
• ffi-bench/tests/frame_compressor_ffi.rs
• ffi-bench/tests/fuzz_interop.rs
• ffi-bench/tests/huff0_ffi.rs
• ffi-bench/tests/match_generator_ffi.rs
• ffi-bench/tests/streaming_encoder_ffi.rs
• zstd/Cargo.toml
• zstd/benches/compare_ffi.rs
• zstd/benches/compare_ffi_sequences.rs
• zstd/benches/support/mod.rs
• zstd/examples/cparams_check.rs
• zstd/examples/cparams_range.rs
• zstd/examples/encode_loop_z000033.rs
• zstd/examples/ffi_compress_z000033.rs
• zstd/examples/trace_fast_kernel.rs
• zstd/src/bit_io/bit_reader_reverse.rs
• zstd/src/bit_io/bit_writer.rs
• zstd/src/cpu_kernel.rs
• zstd/src/decoding/buffer_backend.rs
• zstd/src/decoding/decode_buffer.rs
• zstd/src/decoding/errors.rs
• zstd/src/decoding/exec_sequence_inline.rs
• zstd/src/decoding/flat_buf.rs
• zstd/src/decoding/frame.rs
• zstd/src/decoding/frame_decoder.rs
• zstd/src/decoding/literals_section_decoder.rs
• zstd/src/decoding/prefetch.rs
• zstd/src/decoding/ringbuffer.rs
• zstd/src/decoding/scratch.rs
• zstd/src/decoding/seq_decoder_avx2.rs
• zstd/src/decoding/seq_decoder_scalar.rs
• zstd/src/decoding/sequence_execution.rs
• zstd/src/decoding/sequence_section_decoder.rs
• zstd/src/decoding/simd_copy.rs
• zstd/src/decoding/streaming_decoder.rs
• zstd/src/decoding/user_slice_buf.rs
• zstd/src/dictionary/fastcover.rs
• zstd/src/dictionary/mod.rs
• zstd/src/encoding/blocks/compressed.rs
• zstd/src/encoding/bt/mod.rs
• zstd/src/encoding/cost_model/mod.rs
• zstd/src/encoding/dfast/mod.rs
• zstd/src/encoding/dict_attach.rs
• zstd/src/encoding/fastpath/mod.rs
• zstd/src/encoding/fastpath/scalar.rs
• zstd/src/encoding/fastpath/sse42.rs
• zstd/src/encoding/frame_compressor.rs
• zstd/src/encoding/frame_emit_info.rs
• zstd/src/encoding/frame_header.rs
• zstd/src/encoding/hc/mod.rs
• zstd/src/encoding/incompressible.rs
• zstd/src/encoding/ldm/gear_hash.rs
• zstd/src/encoding/ldm/mod.rs
• zstd/src/encoding/ldm/params.rs
• zstd/src/encoding/ldm/search.rs
• zstd/src/encoding/ldm/table.rs
• zstd/src/encoding/match_generator.rs
• zstd/src/encoding/match_table/helpers.rs
• zstd/src/encoding/match_table/storage.rs
• zstd/src/encoding/mod.rs
• zstd/src/encoding/opt/ldm.rs
• zstd/src/encoding/opt/mod.rs
• zstd/src/encoding/opt/types.rs
• zstd/src/encoding/parameters.rs
• zstd/src/encoding/row/mod.rs
• zstd/src/encoding/sequence_capture.rs
• zstd/src/encoding/simple/fast_kernel/count.rs
• zstd/src/encoding/simple/fast_kernel/hash_table.rs
• zstd/src/encoding/simple/fast_kernel/kernel.rs
• zstd/src/encoding/simple/fast_kernel/mod.rs
• zstd/src/encoding/simple/fast_matcher.rs
• zstd/src/encoding/simple/mod.rs
• zstd/src/encoding/strategy.rs
• zstd/src/encoding/streaming_encoder.rs
• zstd/src/fse/fse_decoder.rs
• zstd/src/fse/fse_encoder.rs
• zstd/src/fse/mod.rs
• zstd/src/histogram.rs
• zstd/src/huff0/huf_cstream.rs
• zstd/src/huff0/huff0_decoder.rs
• zstd/src/huff0/huff0_encoder.rs
• zstd/src/lib.rs
• zstd/src/skippable.rs
• zstd/src/tests/encode_corpus.rs
• zstd/src/tests/fuzz_regressions.rs
• zstd/src/tests/roundtrip_integrity.rs
• zstd/tests/block_splitter_parity.rs
• zstd/tests/corpus_level22_gap.rs
• zstd/tests/cross_validation.rs

💤 Files with no reviewable changes (1)

• zstd/src/tests/encode_corpus.rs

[greptile-apps]

Greptile Summary

• Adds a non-published ffi-bench workspace crate for C FFI benchmarks, parity tests, and interoperability validation.
• Removes C-binding dev dependency usage from the main structured-zstd crate and moves C-dependent tests into ffi-bench.
• Adds bench_internals testing access for moved white-box parity checks.
• Updates repository terminology, docs, examples, comments, and metadata to remove legacy reference-implementation wording.

Confidence Score: 3/5

The refactor has several manifest-level issues that can break or hide intended validation workflows.

The review is based on the changed manifests and moved test/example layout, with targeted runtime checks confirming some Cargo behavior while other checks were limited by local toolchain availability.

ffi-bench/Cargo.toml and zstd/Cargo.toml need attention before relying on the migrated tests, feature controls, and documented example workflows.

T-Rex Logs

What T-Rex did

• Attempted targeted cargo tests corpus_level22_gap and he_level22_ratio, but both were blocked by missing cargo in the environment.
• Ran a targeted repro for cargo builds of structured-zstd examples; both attempts failed because cargo is not installed.
• Ran cargo tree for ffi-bench with no-default-features and confirmed the feature graph resolves while internal features bench_internals, dict_builder, hash, and copy_shape_stats remain enabled.
• Created a temporary integration test to exercise the repro script, ran cargo commands, and cleaned up afterward; one command failed to discover the test target and another ran but reported no tests due to missing cargo.
• Compared before/after artifacts to inspect dependency layout, and attempted a cargo dependency graph, but the run was blocked by missing cargo.
• Performed terminology purge verification, noting the after-state reduces legacy hits but still shows a wrap-aware reference impl in a specific source file.
• Validated C-binding migration results, finding ffi tests discoverable under ffi-bench/tests, while cargo tests still could not run due to missing cargo, making the result inconclusive in this environment.

_{Ran code and verified through T-Rex}

Comments Outside Diff (1)

1. General comment

   Prohibited reference impl terminology remains in changed source comment

   • Bug
     • At head a98714b67430623cf75507d75d0654f105211739, the requested terminology search still finds zstd/src/decoding/buffer_backend.rs:513: /// wrap-aware reference impl; not called on production paths. This is in a changed file and matches the prohibited legacy term reference impl, rather than an allowed prose file:line citation.
   • Cause
     • The repository-wide terminology cleanup missed one shorthand comment in zstd/src/decoding/buffer_backend.rs.
   • Fix
     • Reword the comment to avoid the prohibited shorthand, e.g. wrap-aware checked implementation; not called on production paths, or another neutral phrase that does not use reference impl.

   _{Ran code and verified through T-Rex}

_{Reviews (1): Last reviewed commit: "test: move C-binding unit tests to ffi-b..." | Re-trigger Greptile}

[coderabbitai]

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

zstd/src/encoding/frame_compressor.rs (1)

1085-1091: ⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Replace the remaining legacy “reference implementation” wording.

The PR objective says this terminology was removed from comments, but Lines 1090 and 1312 still use it. Prefer “upstream zstd” here for consistency.

📝 Proposed terminology cleanup

-        // MiB, where the reference implementation's context peaks at
+        // MiB, where upstream zstd's context peaks at
@@
-    /// costs as the reference implementation.
+    /// costs as upstream zstd.

Also applies to: 1308-1312

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@zstd/src/encoding/frame_compressor.rs` around lines 1085 - 1091, Replace the
legacy terminology "the reference implementation" with "upstream zstd" in the
comments to maintain consistency throughout the codebase. Find the comment block
that mentions "reference implementation's context" in the frame compression code
and update this phrase to use "upstream zstd" instead. Also apply the same
terminology change to any other occurrences of "reference implementation" in
related comments (around line 1312 in the same file).

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@zstd/src/encoding/frame_compressor.rs`:
- Around line 1085-1091: Replace the legacy terminology "the reference
implementation" with "upstream zstd" in the comments to maintain consistency
throughout the codebase. Find the comment block that mentions "reference
implementation's context" in the frame compression code and update this phrase
to use "upstream zstd" instead. Also apply the same terminology change to any
other occurrences of "reference implementation" in related comments (around line
1312 in the same file).

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: d70e97cf-b80d-4468-aa03-369a677e06d7

📥 Commits

Reviewing files that changed from the base of the PR and between a98714b and 6c2c637.

📒 Files selected for processing (17)

• .github/scripts/run-benchmarks.sh
• .github/workflows/ci.yml
• c-api/Cargo.toml
• ffi-bench/Cargo.toml
• ffi-bench/tests/encode_corpus_ffi.rs
• ffi-bench/tests/frame_compressor_ffi.rs
• ffi-bench/tests/huff0_ffi.rs
• zstd/Cargo.toml
• zstd/examples/cparams_check.rs
• zstd/examples/cparams_range.rs
• zstd/examples/ffi_compress_z000033.rs
• zstd/examples/section_split.rs
• zstd/src/decoding/errors.rs
• zstd/src/encoding/blocks/compressed.rs
• zstd/src/encoding/frame_compressor.rs
• zstd/src/encoding/levels/fastest.rs
• zstd/src/encoding/match_generator.rs