docs: ARCHITECTURE.md + unified SIMD structural benchmarks#321
Conversation
Add ARCHITECTURE.md — primary entry point documenting the full pipeline data flow from bytes on disk to serialized output. Covers each layer's responsibility, interface types, crate boundaries, buffer lifecycle, and verification strategy. Add structural_detect benchmark comparing separate passes (memchr + ChunkIndex) vs unified SIMD (5-char, 9-char) vs hybrid scalar. Results: 9-char unified SIMD at 3 GiB/s with linear scaling at ~28µs per character — validates extending ChunkIndex into StructuralIndex for format-agnostic structural scanning. Update dev-docs: - DIRECTION.md: add StructuralIndex vision, unified SIMD section, benchmark results - PHASES.md: restructure into 8 phases reflecting current state (Phase 0-1.5 done, Phase 2 is StructuralIndex) - PROVEN_CORE.md: add framer, aggregator, byte_search to module table with proof counts - PROOF_AUDIT.md: update for content_correct oracle proof, close NewlineFramer correctness gap - ZERO_COPY_PIPELINE.md: cross-reference to ARCHITECTURE.md Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
Caution Review failedThe pull request is closed. ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (7)
WalkthroughThis PR adds comprehensive benchmarking infrastructure and developer documentation for structural-character detection in logfwd-core. The main addition is a new Criterion benchmark file ( Possibly related PRs
Comment |
| results.push(unsafe { simd_9char::find_9chars_neon(&block) }); | ||
|
|
||
| #[cfg(target_arch = "x86_64")] | ||
| if is_x86_feature_detected!("avx2") { |
There was a problem hiding this comment.
simd_9char_scan_buffer has no non-AVX2 fallback on x86_64, so on CPUs without AVX2 this benchmark does effectively no scanning work (the loop runs but results never gets a push). That makes unified_9char_simd timings artificially fast on those machines.
Please add an else branch (and ideally a non-x86_64 fallback too) similar to simd_unified_scan_buffer so this benchmark always measures real 9-char detection work.
Summary
dev-docs/ARCHITECTURE.md— primary entry point documenting the full pipeline data flow from input to output, layer interfaces, crate boundaries, buffer lifecycle, and verification strategystructural_detectbenchmark comparing separate passes vs unified SIMD (5-char, 9-char) — validates extending ChunkIndex into StructuralIndex (Unified SIMD StructuralIndex: extend ChunkIndex to detect all structural characters #313)Benchmark results (NEON, ~760KB NDJSON)
Scaling is linear at ~28µs per character. 9-char gives 11x headroom over 1M lines/sec target.
Test plan
cargo bench -p logfwd-core --bench structural_detect🤖 Generated with Claude Code