Skip to content

Split into 5-crate workspace for faster incremental builds#11

Merged
strawgate merged 1 commit into
masterfrom
worktree-wondrous-chasing-hanrahan
Mar 29, 2026
Merged

Split into 5-crate workspace for faster incremental builds#11
strawgate merged 1 commit into
masterfrom
worktree-wondrous-chasing-hanrahan

Conversation

@strawgate
Copy link
Copy Markdown
Owner

Summary

  • Split single-crate project into a Cargo workspace with 5 crates, isolating the heavy datafusion dependency in logfwd-transform so changes to other modules don't trigger its recompilation
  • Updated Dockerfile to copy crates/ instead of src/

Workspace structure

crates/
  logfwd-config/      — YAML config parsing (serde, serde_yaml)
  logfwd-core/        — leaf modules: scanner, batch_builder, cri, compress, otlp, tail, input, diagnostics
  logfwd-transform/   — DataFusion SQL transform (the heavy dep, isolated)
  logfwd-output/      — output sinks (arrow, ureq)
  logfwd/             — binary: main.rs + pipeline_v2 orchestrator

Incremental build times

Change Before (monolith) After (workspace)
scanner.rs ~4s (full crate) 1.4s
otlp_sink.rs ~4s (full crate) ~1s
transform.rs ~4s (full crate) ~2s (only transform + binary)

Test plan

  • cargo test --workspace — all 91 tests pass
  • cargo clippy --workspace -- -D warnings — clean
  • cargo fmt --check — clean
  • No behavioral changes — pure structural refactor

🤖 Generated with Claude Code

Workspace structure:
- logfwd-config: YAML config parsing (serde, serde_yaml)
- logfwd-core: leaf modules — scanner, batch_builder, cri, compress,
  otlp, tail, input, diagnostics (arrow, memchr, notify, zstd)
- logfwd-transform: DataFusion SQL transform (arrow, datafusion, tokio)
- logfwd-output: output sinks (arrow, ureq, uses core + config)
- logfwd: binary crate — main.rs + pipeline_v2 orchestrator

Key incremental build wins:
- Touching scanner.rs: 1.4s (only core + downstream, no datafusion)
- Touching otlp_sink.rs: ~1s (only output + binary)
- Touching transform.rs: only transform + binary recompile

All 91 tests pass, clippy clean, fmt clean.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@strawgate strawgate merged commit 7c42900 into master Mar 29, 2026
1 check failed
@strawgate strawgate deleted the worktree-wondrous-chasing-hanrahan branch March 29, 2026 06:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant