Skip to content

fix: normalize conflict struct columns in OTLP sink before encoding#832

Merged
strawgate merged 2 commits into
masterfrom
copilot/fix-otlp-sink-data-loss
Apr 3, 2026
Merged

fix: normalize conflict struct columns in OTLP sink before encoding#832
strawgate merged 2 commits into
masterfrom
copilot/fix-otlp-sink-data-loss

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented Apr 3, 2026

OtlpSink::encode_batch silently dropped any Arrow Struct-typed column via DataType::Struct(_) => continue, causing data loss for fields with mixed types (which the conflict schema mechanism encodes as Struct).

Changes

  • logfwd-arrow: Moved conflict_schema module here from logfwd-transform so it's accessible to output crates without a circular dependency
  • logfwd-transform: Re-exports logfwd_arrow::conflict_schema for backward compatibility
  • logfwd-output: Added logfwd-arrow dependency; calls normalize_conflict_columns on the batch before the encoding loop so conflict Struct columns are flattened to Utf8 rather than skipped
// Before: silently dropped
DataType::Struct(_) => continue,

// After: normalize first, then encode as Utf8
let batch = normalize_conflict_columns(&batch)?;
// ... encoding loop now sees Utf8 columns

Copilot AI linked an issue Apr 3, 2026 that may be closed by this pull request
3 tasks
Copilot AI changed the title [WIP] Fix OTLP sink to handle conflict struct columns fix: normalize conflict struct columns in OTLP sink before encoding Apr 3, 2026
Copilot AI requested a review from strawgate April 3, 2026 07:39
@strawgate strawgate marked this pull request as ready for review April 3, 2026 14:54
@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented Apr 3, 2026

Important

Review skipped

Bot user detected.

To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository YAML (base), Organization UI (inherited)

Review profile: ASSERTIVE

Plan: Free

Run ID: 065b9674-d0d5-4338-86aa-f384db650fad

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • ✅ Review completed - (🔄 Check again to review again)

Comment @coderabbitai help to get the list of available commands and usage tips.

Copilot AI and others added 2 commits April 3, 2026 10:25
Co-authored-by: strawgate <6384545+strawgate@users.noreply.github.com>
@strawgate strawgate force-pushed the copilot/fix-otlp-sink-data-loss branch from 0e4e668 to 19d8173 Compare April 3, 2026 15:26
@strawgate strawgate merged commit c9e6393 into master Apr 3, 2026
7 checks passed
@strawgate strawgate deleted the copilot/fix-otlp-sink-data-loss branch April 3, 2026 16:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: OTLP sink silently drops conflict struct columns — data loss

2 participants