Benchmark harness: hot-path micro-benches + yip-vs-WireGuard netem - #1
Merged
Conversation
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add run-compare.sh and a comparison_under_netem_loss test that run yip and kernel WireGuard side-by-side under identical tc netem loss profiles (0 1 3 5 10%) and emit a headline table. At 10% injected loss the real measured result is yip_loss=1% vs wg_loss=20%, confirming that RaptorQ FEC recovers most loss that WireGuard passes through unchanged. Saves the combined table to RESULTS.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Add crates/yip-bench/README.md: curated results writeup (survives harness re-runs that overwrite RESULTS.md). Covers hot-path ns/op table with honesty note on aead_open (seal+open, not open-only), the yip-vs-WireGuard netem comparison table, environment label (kernel 6.18, AMD Ryzen 5 7640U), and deferred items. - Move yip-wire and yip-transport to [dev-dependencies] in crates/yip-bench/Cargo.toml (cargo-shear: bench-only crates). - Add bench-smoke CI job (hermetic, fast: --sample-size 10) and netem-comparison CI job (sudo-gated, installs wireguard-tools, honesty guard on yip-only skips) to integration.yml. - CHANGELOG.md: add yip-bench entry under ### Added. - README.md: mark M7 (benchmark harness) done; update Next pointer. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A bare cargo bench also runs the lib's default harness, which rejects Criterion's --warm-up-time flag. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Measure file-copy (scp/TCP) throughput over the yip tunnel vs kernel WireGuard under tc netem packet loss (0/5/10 %). The throughput thesis: yip's RaptorQ FEC masks loss from TCP so throughput holds, while WireGuard's TCP sees real retransmits and collapses under loss. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The netns benchmark suite built and ran a *debug* yipd (`cargo build -p yipd`) and compared it against in-kernel WireGuard — an apples-to-oranges comparison. yipd's RaptorQ data path is ~75x slower compiled without optimizations (encode ~1916 us/object debug vs ~25.6 us release; the GF(256) matrix math dominates), which throttled the egress thread to ~6 Mbit/s and inflated per-packet latency by ~2 ms. That produced the bogus "yip ~0.5 MB/s, ~18 ms RTT, +8 ms FEC premium" figures — all build-mode artifacts, not FEC cost. Root cause was found by instrumenting the loop (zero ENOBUFS/deframe/open errors, zero TCP retransmits — so not self-inflicted loss) then timing encode in debug vs release. Confirmed end-to-end: release yipd does 273 Mbit/s iperf3 (0% UDP loss) and adds only ~0.2 ms RTT over WireGuard. Fix: build/run yipd `--release` everywhere (run-compare.sh, run-scp-compare.sh, run-yip-netem.sh, netem_bench.rs's three paths, and the CI integration job), and re-measure. Corrected numbers (release, kernel 6.18, Ryzen 5 7640U): netem loss sweep: at 10% injected, yip 1% effective vs WG 17%, RTT 10.54 vs 10.34 ms (FEC premium ~0.2 ms, not 8 ms). scp throughput: 0% loss yip 13.8 / WG 35.2 MB/s; 5% yip 2.44 / WG 0.37; 10% yip 1.02 / WG 0.16 MB/s — yip ~6x WireGuard under loss. Updated crates/yip-bench/README.md (netem + scp sections, build note, caveats), the root README M7 bullet, and RESULTS.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Expands the netem comparison beyond yip-vs-WireGuard into a full matrix, each contender's netns setup spiked and verified under sudo before wiring. New harnesses (all SKIP cleanly if a tool is absent; yip + WireGuard are the must-run core; every external VPN built/run release / in-kernel for fairness): - run-fec-compare.sh + udp_tx.py/udp_rx.py — the FEC-vs-FEC headline. A pure-UDP sequenced blaster (iperf3 can't traverse UDPspeeder — it needs a TCP control channel) measures delivered-loss across bare-link (no FEC) vs UDPspeeder (Reed-Solomon f20:10) vs yip (RaptorQ) under a netem sweep. UDPspeeder is yip's closest competitor — both add FEC over UDP. Measured (10% loss): bare 90.4% delivered, UDPspeeder 100%, yip 99.0% — both FECs erase the loss the bare link drops. UDPspeeder built from refrences/UDPspeeder to .bench-tools/speederv2 (git-ignored; located via SPEEDERV2 env / .bench-tools / PATH). - run-iperf-compare.sh — iperf3 TCP throughput + ping latency/loss across the full-IP tunnels yip / WireGuard / OpenVPN / n2n under a netem sweep. Measured: on a clean link in-kernel WireGuard wins raw speed (~1 Gbit/s) and the userspace tunnels cluster (yip 167, OpenVPN 173, n2n 191 Mbit/s); under loss yip pulls ahead of every no-FEC tunnel — at 10% loss yip holds 9.3 Mbit/s + 6% effective loss while WG/n2n collapse to ~1.3-1.7 and OpenVPN's TCP times out. OpenVPN: static-key AES-256-CBC p2p TUN (GCM is TLS-only; --secret needs --allow-deprecated-insecure-static-crypto on 2.7). n2n: v3 supernode + 2 edges, one TAP data plane that serves both L2 and L3 (measured once). Two root-gated tests (udp_loss_recovery_comparison, iperf_throughput_comparison) + CI steps whose honesty guards fail on a "needs root" skip but allow tool-absent contender skips. Plan doc records every verified setup command. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The contender used static-key AES-256-CBC, which understates OpenVPN — static-key mode has no AEAD/GCM, but GCM (with DCO) is its real deployed data channel. Switch run-iperf-compare.sh to TLS peer-fingerprint mode (self-signed EC certs, no PKI) with --data-ciphers AES-256-GCM; SKIP if openssl is absent. Spiked first: clean zero-delay link reaches >1 Gbit/s (vs 816 Mbit/s CBC). Re-measured the matrix; the under-loss story is unchanged and if anything cleaner: at 10% loss yip holds 8.9 Mbit/s + 2% effective loss while WireGuard/OpenVPN/n2n collapse to ~1-1.4 Mbit/s with 6-32% effective loss. Updated the README tables, contender description, and interpretation. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
vxfemboy
added a commit
that referenced
this pull request
Jun 30, 2026
* Spec the data-plane throughput pass Incremental, measurement-driven optimization milestone for sub-project #1: raise single-core throughput (clean-link weak spot) without changing the wire format. Profile-first gate, then attack the dominant per-packet cost (FEC encode ~25us: bypass RaptorQ when repair=0, amortize over larger Bulk objects), then I/O batching (GSO/sendmmsg/recvmmsg, buffer reuse) and finally wire yipd onto yip-io. The unified io_uring busy-poll rewrite stays explicitly deferred. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Plan the data-plane throughput pass Six tasks: (1) per-stage pipeline profile gate; (2) bypass RaptorQ encode when repair=0, guarded by a byte-identical round-trip test; (3) batched send/recv (sendmmsg/recvmmsg) on DataPlaneIo; (4) egress buffer reuse + batch send in yipd; (5) wire yipd onto yip-io + recvmmsg ingress + socket buffers; (6) end-to-end before/after measurement. Bulk amortization + GSO are stretch. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Profile the egress/ingress pipeline per stage * Replace as-cast with TryFrom in pipeline_profile * Bypass RaptorQ encode when zero repair symbols are requested Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Add 2-symbol byte-identical test; dedupe OTI on bypass fallback - Add zero_repair_bypass_byte_identical_two_full_symbols: mirrors the existing byte-identical test with 2400-byte input (2 × symbol_size 1200, no remainder), confirming the bypass loop runs twice with no padding branch. - Fix misleading comment on zero_repair_bypass_is_byte_identical_to_encoder: 1200 bytes is exactly one full source symbol, not "> one symbol". - Dedupe OTI construction on the repair==0 path: hoist ObjectTransmissionInformation::with_defaults above the repair==0 branch so the same oti is reused by both the bypass gate check and the fallback Encoder::new call, eliminating the duplicate construction. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Add batched send/recv (sendmmsg/recvmmsg) to DataPlaneIo Adds MAX_DATAGRAM_BATCH=64 and MAX_WIRE_DATAGRAM=2048 module consts. Extends the DataPlaneIo trait with send_batch/recv_batch default impls (loop send / single recv) so IoUringIo compiles unchanged. Overrides both in PlainIo using libc::sendmmsg/recvmmsg with MSG_NOSIGNAL / MSG_WAITFORONE; all unsafe blocks carry SAFETY comments. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Pin libc exactly; guard default recv_batch against empty slices - Change libc from "0.2" to exact version "0.2.186" (already in Cargo.lock) - Add empty-slice guard to default recv_batch() to prevent panic when bufs or lens arrays are empty (PlainIo override already guards) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Egress: reuse buffers and batch-send a packet's symbols Replace per-symbol Vec allocation + per-symbol send() with a thread-owned arena of pre-allocated Vec<u8> slots (cleared and refilled each packet) and a single PlainIo::send_batch() call (sendmmsg) per TUN packet. After the first packet, zero heap allocation occurs per symbol; the only remaining per-packet allocation is a small Vec<&[u8]> of pointer-sized slice views. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Ingress: recvmmsg batch via yip-io; size socket buffers Replace the per-datagram udp_rx.recv() in the ingress thread with PlainIo::recv_batch() (recvmmsg/MSG_WAITFORONE), draining up to 64 datagrams per syscall. Batch buffers (64 × 2 KiB) are heap-allocated once before the loop and reused. The per-datagram deframe→decode→open→ write_frame pipeline is unchanged; it now runs inside a for-i-in-0..n inner loop. Add yip_io::set_socket_buffers() — a safe public fn backed by a pair of setsockopt(SO_SNDBUF/SO_RCVBUF) unsafe blocks with SAFETY comments — so yipd (#![forbid(unsafe_code)]) can raise both buffers to 4 MiB after connect(). Add a yip-io unit test that calls it on a loopback socket and asserts success (kernel may clamp the exact value). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Use TryFrom for socklen_t; forbid unsafe_code in yipd Replace numeric casts with TryFrom conversions for socket buffer size in yip-io's set_socket_buffers function to comply with lint rules. Add #![forbid(unsafe_code)] to yipd binary crate since it contains no unsafe code. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Measure and record the data-plane throughput pass Clean-link single-stream TCP ~220-285 Mbit/s — no regression vs the pre-pass baseline. The honest verdict: the FEC-encode bypass is dormant because the controller's repair_count floors at max(1) and never requests zero repair, so every packet still runs the encoder and carries a redundant repair symbol. That floor is load-bearing until the daemon feeds observed loss back to the controller (deferred ARQ/feedback) — so the clean-link win (skip encode + halve datagrams) is unlocked by the adaptive feedback loop, not this pass alone. This pass shipped the plumbing (batched sendmmsg/recvmmsg, reused buffers, 4 MiB socket buffers, a ready-and-tested zero-repair bypass). Recorded in the bench README + CHANGELOG. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Correct misleading comment on the source-symbol padding branch --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
A
yip-benchdev-only crate that validates yip's thesis — low latency, and FEC that recovers loss plain tunnels pass through.(a) Criterion hot-path micro-benchmarks (
benches/hotpath.rs):(The
aead_openbench measures seal+open since the replay window rejects a repeated counter; open alone ≈ ~2µs — labeled honestly in the results doc.)(b)
tc netemcomparison vs kernel WireGuard (sudo-gated, runs in CI): both tunnels in separate netns over identical loss sweeps.yip's RaptorQ FEC recovers the loss WireGuard passes straight through (X² law: yip ≈ p², WG ≈ p), at an ~8ms RTT premium (~18ms vs ~10ms) from FEC encoding. The WireGuard comparison is apples-to-apples (verified config, no MTU mismatch, warmed handshake).
Honesty / scope
crates/yip-bench/README.md): iperf3 throughput, L2 contenders (n2n/ZeroTier/OpenVPN), TCP-under-loss.yip-benchispublish=false; criterion + the path deps are dev-only — no shipped crate depends on it.CI
bench-smoke(hermetic): compiles + quick-runs every Criterion group.netem-comparison(sudo): runs the yip-vs-WireGuard sweep with an honesty guard. Both verified green on a GitHub runner.Full writeup:
crates/yip-bench/README.md. Plan:docs/superpowers/plans/2026-06-30-yip-bench-harness.md.🤖 Generated with Claude Code