fix(scanner): decode JSON escape sequences in string values (#410)

[Copilot]

Scanner stored raw JSON escape bytes in string values. The output layer then re-escaped backslashes, turning \u00e9 → \\u00e9 — syntactically valid but semantically wrong JSON. Affects all escape sequences: \", \\, \/, \b, \f, \n, \r, \t, \uXXXX.

Changes

Scanner escape decoding (json_scanner.rs)

• Added decode_json_escapes covering all RFC 8259 §7 escapes including \uXXXX surrogate pairs
• Fast path: memchr(b'\\', raw) check — strings without backslashes pass through zero-copy unchanged
• Slow path: decode into a scratch buffer allocated once per scan_streaming call, reused across lines via clear()

ScanBuilder trait (scanner.rs)

• Added append_decoded_str_by_idx with default delegation to append_str_by_idx — signals that the value is decoded and may not be a subslice of the input buffer

StreamingBuilder override (streaming_builder.rs)

• StreamingBuilder uses zero-copy views (offset into input buffer), so decoded bytes can't use offset_of
• Override stores decoded strings as Vec<u8> in a separate decoded_strs list per field
• finish_batch merges zero-copy try_append_view entries with decoded append_value entries by row number

Conformance / compliance test updates

• Removed !actual.contains('\\') guard in oracle comparison — scanner now matches sonic-rs for all strings including those with escapes
• Updated compliance_special_chars_in_values and compliance_escaped_quotes_in_strings to expect decoded characters
• Added \uXXXX BMP code points to arb_json_string proptest generator

// Before: raw bytes passed through, double-escaped downstream
builder.append_str_by_idx(idx, &buf[val_start..val_end]);

// After: decode escapes, use appropriate append method
if memchr::memchr(b'\\', raw).is_some() {
    decode_json_escapes(raw, scratch);
    builder.append_decoded_str_by_idx(idx, scratch);
} else {
    builder.append_str_by_idx(idx, raw);
}

[strawgate]

@coderabbitai review

[coderabbitai]

✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[coderabbitai]

Warning

Rate limit exceeded

@strawgate has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 25 minutes and 28 seconds before requesting another review.

Your organization is not enrolled in usage-based pricing. Contact your admin to enable usage-based pricing to continue reviews beyond the rate limit, or try again in 25 minutes and 28 seconds.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository YAML (base), Organization UI (inherited)

Review profile: ASSERTIVE

Plan: Pro

Run ID: 1b3768c9-7299-49f1-a277-a1b9196b477a

📥 Commits

Reviewing files that changed from the base of the PR and between 26cee6e and 880d27f.

📒 Files selected for processing (7)

• crates/logfwd-arrow/src/scanner.rs
• crates/logfwd-arrow/src/streaming_builder.rs
• crates/logfwd-core/src/json_scanner.rs
• crates/logfwd-core/src/scanner.rs
• crates/logfwd-core/tests/compliance_data.rs
• crates/logfwd-core/tests/scanner_conformance.rs
• crates/logfwd-test-utils/src/json.rs

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}