a learning-oriented log forwarder built with Rust

---

FastForward is a research and learning project exploring how far you can push a log forwarding pipeline with modern Rust tooling. It tails files, parses JSON and CRI logs with portable SIMD, transforms with SQL, and ships to OTLP collectors.

The documentation site has interactive guides that explain how each piece works — from SIMD parsing to backpressure to checkpoint ordering — with live simulations you can play with.

Note: The Cargo package is named ffwd; the installed CLI binary is ff.

Try it

# Build from source (Rust 1.89+)
git clone https://github.com/strawgate/fastforward.git && cd fastforward
cargo build --release -p ffwd

# Generate some test data
./target/release/ff generate-json 100000 logs.json

# config.yaml
input:
  type: file
  path: logs.json
  format: json

transform: |
  SELECT level, message, status, duration_ms
  FROM logs
  WHERE level = 'ERROR' AND duration_ms > 50

output:
  type: stdout
  format: console

./target/release/ff run --config config.yaml

Only error records with slow durations make it through — everything else is filtered by the SQL transform.

For one-off command-line sends, keep a destination-only config and pipe data into ff:

# destination.yaml
output:
  type: otlp
  endpoint: http://127.0.0.1:4318/v1/logs

cat logs.json | ./target/release/ff send --config destination.yaml --format json --service checkout

What makes it interesting

log files → SIMD parse → Arrow RecordBatch → DataFusion SQL → OTLP → your collector

• Portable SIMD parsing — 10 broadcast-compare ops per 64-byte block, one pass per buffer, runs on x86_64 and ARM64
• Zero-copy Arrow pipeline — StringViewArray stores views into the read buffer; string data isn't copied from scanner to RecordBatch
• SQL transforms — every batch runs through a DataFusion SQL query with built-in UDFs like regexp_extract(), grok(), and geo_lookup()
• Formal verification — the parsing core is verified with Kani (bounded model checking), state machines with TLA+, and SIMD conformance with proptest
• Single static binary — ~15 MB, no JVM, no Python, no runtime dependencies

Documentation

The docs are the best way to understand FastForward:

• Quick Start — install, run your first pipeline, ship logs
• SQL Transforms — filter, reshape, extract, join
• Configuration Reference — every YAML field, input/output type, UDF

The "Understand It" section of the docs has interactive guides with live simulations:

• Tailing — watch file rotation and truncation handling live
• SIMD Scanner — step through JSON parsing, toggle field pushdown
• Backpressure — slow the output and watch pressure cascade back
• Columnar Storage — row vs column layout, why Arrow makes SQL fast
• Checkpoint Ordering — out-of-order ACKs and the committed watermark

Contributing

This is a learning project — contributions, questions, and experiments are welcome.

• Quick Start for contributors — task routing and developer docs
• CONTRIBUTING.md — PR process and pre-commit checks
• DEVELOPING.md — build, test, lint, bench commands

just build    # release binary
just test     # run tests
just lint     # fmt + clippy + toml check
just ci       # lint + test (run before pushing)