fix: increment parse_errors for invalid JSON lines in passthrough JSON mode

[Copilot]

Invalid JSON lines (e.g. not json at all) in Format::Json mode silently became all-null records — the scanner produces an empty row when a line doesn't start with {, and nothing incremented parse_errors.

Changes

FormatProcessor redesign (logfwd-io/src/format.rs)

• Split the unit Passthrough variant into two struct variants:
  • Passthrough { stats } — raw format, forwards verbatim, never counts errors
  • PassthroughJson { stats } — JSON format, forwards verbatim and increments parse_errors for any non-empty line whose first non-whitespace byte is not {
• Added passthrough(stats) and passthrough_json(stats) constructors (consistent with cri()/auto() pattern)
• Added count_json_parse_errors() helper that scans a chunk line-by-line without copying

Pipeline wiring (logfwd/src/pipeline.rs)

• Format::Json → passthrough_json(stats) (counts errors)
• Format::Raw → passthrough(stats) (no validation — non-JSON is expected)

Call sites updated (framed.rs, allocation_churn.rs) to use passthrough(stats) constructor.

Behavior

Lines are still forwarded to the scanner unchanged (option b from the issue spec). The parse_errors counter now increments per invalid line, making data-quality issues visible in diagnostics without dropping any data.

# Before: parse_errors stays 0, record emitted with all fields null
# After:  parse_errors = 1, same record emitted (data preserved, error surfaced)
echo 'not json at all' | logfwd ...

[strawgate]

@coderabbitai please do a full review

[coderabbitai]

@strawgate Sure, kicking off a full review now!

---

🧠 Learnings used

Learnt from: CR
Repo: strawgate/memagent PR: 0
File: coderabbit-custom-pre-merge-checks-unique-id-file-non-traceable-F7F2B60C-1728-4C9A-8889-4F2235E186CA.txt:0-0
Timestamp: 2026-04-03T01:38:58.608Z
Learning: Applies to crates/logfwd-core/src/**/*.rs : New async pipeline code must have proptest coverage for: arbitrary event ordering; acks out of order; drain while in-flight

---

Learnt from: CR
Repo: strawgate/memagent PR: 0
File: AGENTS.md:0-0
Timestamp: 2026-04-02T07:03:19.858Z
Learning: Applies to **/{scanner,*parser*,builder*,encoder*}.rs : Avoid per-line heap allocations in hot paths: scanner.rs, format parser, CRI parser, builders, OTLP encoder, and compression code

✅ Actions performed

Full review triggered.

[coderabbitai]

Caution

Review failed

Pull request was closed or merged during review

Walkthrough

The FormatProcessor enum was refactored to carry Arc<ComponentStats> in both Passthrough and new PassthroughJson variants. Two constructors were added: passthrough() and passthrough_json(). A helper function count_json_parse_errors() scans lines and increments error stats for lines not matching JSON object format. The make_format function now differentiates between JSON and Raw formats by selecting distinct passthrough processors. Test usages were updated to call the new constructors instead of using the variant directly.

Possibly related PRs

• refactor: move format handling and framing out of pipeline into input layer #581: Introduced the original FormatProcessor enum and its process_lines/reset behavior that this PR directly modifies by adding stats-carrying variants and JSON error counting.
• refactor: pipeline bypasses format.rs — raw bytes to scanner (#303, #335) #399: Refactors format-handling logic in pipeline.rs and affects how FormatProcessor variants are constructed and used downstream.

---

Caution

Pre-merge checks failed

Please resolve all errors before merging. Addressing warnings is optional.

• Ignore

❌ Failed checks (1 error, 2 warnings)

Check name	Status	Explanation	Resolution
High-Quality Rust Practices	❌ Error	Logic bug in count_json_parse_errors: whitespace-only lines incorrectly increment parse_errors when first_nonws is None, violating documented behavior to ignore empty/whitespace-only lines.	Restructure condition to if let Some(p) = line.iter().position(...) { if line[p] != b'{' { parse_errors += 1 } } and add regression tests for whitespace-only inputs.
Documentation Thoroughly Updated	⚠️ Warning	PR adds public API functions with doc comments but documentation updates incomplete: CONFIG_REFERENCE.md missing json format parse_errors behavior, DESIGN.md lacks ADR for PassthroughJson split, DEVELOPING.md lacks whitespace edge case documentation, critical bug in count_json_parse_errors incorrectly counts whitespace-only lines as parse errors.	Fix count_json_parse_errors bug, update CONFIG_REFERENCE.md for parse_errors behavior, add ADR to DESIGN.md, document whitespace edge case in DEVELOPING.md, add regression test.
Maintainer Fitness	⚠️ Warning	PR modifies hot-path format processing without benchmark results and has unaddressed code review feedback identifying a whitespace-only line handling bug.	Fix count_json_parse_errors to properly handle whitespace-only lines, run just bench before/after, and update PR description with risk surface and test coverage details.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Formal Verification Coverage	✅ Passed	PR contains no logfwd-core modifications; new public functions in logfwd-io are trivial constructors exempt from verification requirements.
Crate Boundary And Dependency Integrity	✅ Passed	PR maintains crate boundary integrity with no new crates, no unsafe code, and correct dependency hierarchy preserved.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}