Wire v2 Arrow pipeline: config → scan → DataFusion → output

[strawgate]

Summary

• Connects all v2 Arrow pipeline components into a working pipeline driven by YAML config
• New pipeline_v2.rs: Pipeline::from_config() + run() main loop wiring file tailing → JSON/CRI parsing → Scanner → Arrow RecordBatch → DataFusion SQL → OutputSink
• New CLI flags: --config, --validate, --dry-run for v2 mode; --blackhole for fake OTLP collector
• Fix: strip type suffixes in scan_config() so field pushdown works with typed SQL names (level_str → level)
• Fix: handle empty RecordBatches in batch_builder.rs without panic

Performance (5M lines × 176 bytes × 7 JSON fields, single core)

JSON stdout output:
  SELECT * (passthrough)              2,907,415 l/s   343 ns/l   487 MB/s
  WHERE level != DEBUG (25% drop)     2,702,530 l/s   370 ns/l   453 MB/s
  WHERE level = ERROR (75% drop)      3,060,090 l/s   326 ns/l   513 MB/s
  SELECT 3 cols WHERE dur > 200       3,837,732 l/s   260 ns/l   643 MB/s

OTLP protobuf → blackhole collector:
  SELECT * → OTLP (no compress)       2,643,890 l/s   378 ns/l   443 MB/s
  WHERE != DEBUG → OTLP               2,919,754 l/s   342 ns/l   489 MB/s
  SELECT * → OTLP + zstd              3,070,802 l/s   325 ns/l   515 MB/s
  WHERE != DEBUG → OTLP + zstd        2,858,889 l/s   349 ns/l   479 MB/s

Test plan

• All 107 tests pass (96 existing + 11 new pipeline_v2 tests)
• --config config.yaml --validate validates and exits
• --config config.yaml --dry-run constructs pipeline and exits
• --config config.yaml processes JSON file with SELECT * passthrough
• WHERE filtering produces correct row counts (25K/100K for ERROR-only)
• Column projection outputs only selected fields
• Type preservation: integers stay integers in JSON output
• OTLP output to blackhole collector works end-to-end
• Diagnostics endpoints (/health, /metrics, /api/pipelines) report correct stats

🤖 Generated with Claude Code