Data-plane throughput pass: batched I/O + zero-repair FEC bypass - #2
Merged
Conversation
Incremental, measurement-driven optimization milestone for sub-project #1: raise single-core throughput (clean-link weak spot) without changing the wire format. Profile-first gate, then attack the dominant per-packet cost (FEC encode ~25us: bypass RaptorQ when repair=0, amortize over larger Bulk objects), then I/O batching (GSO/sendmmsg/recvmmsg, buffer reuse) and finally wire yipd onto yip-io. The unified io_uring busy-poll rewrite stays explicitly deferred. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Six tasks: (1) per-stage pipeline profile gate; (2) bypass RaptorQ encode when repair=0, guarded by a byte-identical round-trip test; (3) batched send/recv (sendmmsg/recvmmsg) on DataPlaneIo; (4) egress buffer reuse + batch send in yipd; (5) wire yipd onto yip-io + recvmmsg ingress + socket buffers; (6) end-to-end before/after measurement. Bulk amortization + GSO are stretch. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add zero_repair_bypass_byte_identical_two_full_symbols: mirrors the existing byte-identical test with 2400-byte input (2 × symbol_size 1200, no remainder), confirming the bypass loop runs twice with no padding branch. - Fix misleading comment on zero_repair_bypass_is_byte_identical_to_encoder: 1200 bytes is exactly one full source symbol, not "> one symbol". - Dedupe OTI construction on the repair==0 path: hoist ObjectTransmissionInformation::with_defaults above the repair==0 branch so the same oti is reused by both the bypass gate check and the fallback Encoder::new call, eliminating the duplicate construction. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds MAX_DATAGRAM_BATCH=64 and MAX_WIRE_DATAGRAM=2048 module consts. Extends the DataPlaneIo trait with send_batch/recv_batch default impls (loop send / single recv) so IoUringIo compiles unchanged. Overrides both in PlainIo using libc::sendmmsg/recvmmsg with MSG_NOSIGNAL / MSG_WAITFORONE; all unsafe blocks carry SAFETY comments. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Change libc from "0.2" to exact version "0.2.186" (already in Cargo.lock) - Add empty-slice guard to default recv_batch() to prevent panic when bufs or lens arrays are empty (PlainIo override already guards) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace per-symbol Vec allocation + per-symbol send() with a thread-owned arena of pre-allocated Vec<u8> slots (cleared and refilled each packet) and a single PlainIo::send_batch() call (sendmmsg) per TUN packet. After the first packet, zero heap allocation occurs per symbol; the only remaining per-packet allocation is a small Vec<&[u8]> of pointer-sized slice views. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace the per-datagram udp_rx.recv() in the ingress thread with PlainIo::recv_batch() (recvmmsg/MSG_WAITFORONE), draining up to 64 datagrams per syscall. Batch buffers (64 × 2 KiB) are heap-allocated once before the loop and reused. The per-datagram deframe→decode→open→ write_frame pipeline is unchanged; it now runs inside a for-i-in-0..n inner loop. Add yip_io::set_socket_buffers() — a safe public fn backed by a pair of setsockopt(SO_SNDBUF/SO_RCVBUF) unsafe blocks with SAFETY comments — so yipd (#![forbid(unsafe_code)]) can raise both buffers to 4 MiB after connect(). Add a yip-io unit test that calls it on a loopback socket and asserts success (kernel may clamp the exact value). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace numeric casts with TryFrom conversions for socket buffer size in yip-io's set_socket_buffers function to comply with lint rules. Add #![forbid(unsafe_code)] to yipd binary crate since it contains no unsafe code. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Clean-link single-stream TCP ~220-285 Mbit/s — no regression vs the pre-pass baseline. The honest verdict: the FEC-encode bypass is dormant because the controller's repair_count floors at max(1) and never requests zero repair, so every packet still runs the encoder and carries a redundant repair symbol. That floor is load-bearing until the daemon feeds observed loss back to the controller (deferred ARQ/feedback) — so the clean-link win (skip encode + halve datagrams) is unlocked by the adaptive feedback loop, not this pass alone. This pass shipped the plumbing (batched sendmmsg/recvmmsg, reused buffers, 4 MiB socket buffers, a ready-and-tested zero-repair bypass). Recorded in the bench README + CHANGELOG. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This was referenced Jul 1, 2026
vxfemboy
added a commit
that referenced
this pull request
Jul 3, 2026
#32) Two design specs for the throughput roadmap (design only, no code): - multicore-sharding-design.md (#10): per-peer engine sharding — N independent single-thread DataPlane engines (one per core), peers assigned by hash, each with its own SO_REUSEPORT UDP socket + IFF_MULTI_QUEUE TUN queue. No shared state/locks; preserves the Phase A per-flow latency; reuses the driver loops verbatim; `engines=N` (engines=1 == today). Scales AGGREGATE throughput across peers; depends on the multi-peer data plane (sub-project #2), so a no-op for today's single peer. Single-flow crypto worker-pool documented as deferred regime B. - fec-object-batching-design.md: batch N sealed packets into one RaptorQ object to amortize the ~24µs Encoder::new setup (~24/N µs/packet). Container framing + derived AEAD counters (no wire change); plaintext accumulator sealed consecutively at flush (keeps LossDetector counters contiguous); Bulk-only, N=8/1ms. Honest scope: helps only the repair>0 (lossy) path — clean links already bypass the encoder. Both settle the "how do we reach WireGuard throughput" question: it's CPU/single- thread bound (FEC encode + one core), NOT I/O — so multi-core is the primary lever, batching compounds it on lossy links. Follow-ups filed as #27-#31. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
vxfemboy
added a commit
that referenced
this pull request
Jul 5, 2026
…roject #2, 2a) (#33) * docs: design spec for multi-peer data plane + self-certifying addresses (2a) First milestone of sub-project #2 (control plane). Turns yipd from a single connected peer into a static N-peer mesh: key-derived self-certifying inner addresses (BLAKE2s(pubkey) -> fd::/128), a peer-list config, a thin PeerManager over per-peer DataPlanes, an addressed on_udp/EgressDatagram seam (recvfrom/sendto, which also unblocks multi-core #10), and an in-loop lazy WireGuard-style handshake. Discovery (2c), NAT traversal/relay (2b), subnets, and multi-core (#10) are explicit non-goals. Approved in brainstorming; pending spec review before writing-plans. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: implementation plan for multi-peer data plane (2a) Six-task TDD plan: (1) key-derived address helper, (2) peer-list config, (3) the addressed socket seam (recvfrom/sendto, on_udp(src) + EgressDatagram.dst) as a no-behavior-change refactor gated on the single-peer netns suite under both drivers, (4) handshake step-functions, (5) PeerManager (routing/demux + in-loop lazy handshake over per-peer DataPlanes) + tunnel wiring, (6) 3-peer netns triangle + CI. Each task ends on an independently testable deliverable; global constraints (forbid unsafe in yipd, no `as` casts, no wire regression) carried. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This was referenced Jul 6, 2026
vxfemboy
added a commit
that referenced
this pull request
Jul 7, 2026
Milestone 2c: decentralized discovery (completes sub-project #2)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A measurement-driven optimization pass on the yipd data-plane hot path. Spec + plan committed; executed as 6 reviewed tasks.
What landed (no wire-format change — netns ping 3/3 throughout)
sendmmsg; ingress reads bursts viarecvmmsg(MSG_WAITFORONE). yipd now uses yip-io'sPlainIoinstead of a rawUdpSocket.SO_SNDBUF/SO_RCVBUFvia a yip-ioset_socket_buffershelper. yipd is now#![forbid(unsafe_code)];libcpinned exactly.yip-transport) — skips the ~24 µsEncoder::newsolve, emitting source symbols byte-identically to the encoder (verified across 13 object sizes in the final review).Honest verdict (see
crates/yip-bench/README.md)Clean-link single-stream TCP ~220–285 Mbit/s — no regression. The FEC-encode bypass is dormant: the controller's
repair_countfloors atmax(1), so it never requests zero repair. That floor is load-bearing until the daemon feeds observed loss back to the controller (deferred ARQ/feedback) — dropping repair to zero without that would disable FEC entirely. So the clean-link win (skip encode + halve per-packet datagram count) is unlocked by the adaptive loss-feedback loop (the next milestone), not this pass alone. This pass shipped the plumbing and a ready, tested bypass.Review
Per-task reviews + a final whole-branch opus review: READY TO MERGE, no Critical/Important findings; cumulative
unsafe(sendmmsg/recvmmsg/setsockopt) verified sound, no network-input panic paths.🤖 Generated with Claude Code