docs: multi-core sharding (#10) + FEC object batching design specs - #32
Merged
Conversation
Two design specs for the throughput roadmap (design only, no code): - multicore-sharding-design.md (#10): per-peer engine sharding — N independent single-thread DataPlane engines (one per core), peers assigned by hash, each with its own SO_REUSEPORT UDP socket + IFF_MULTI_QUEUE TUN queue. No shared state/locks; preserves the Phase A per-flow latency; reuses the driver loops verbatim; `engines=N` (engines=1 == today). Scales AGGREGATE throughput across peers; depends on the multi-peer data plane (sub-project #2), so a no-op for today's single peer. Single-flow crypto worker-pool documented as deferred regime B. - fec-object-batching-design.md: batch N sealed packets into one RaptorQ object to amortize the ~24µs Encoder::new setup (~24/N µs/packet). Container framing + derived AEAD counters (no wire change); plaintext accumulator sealed consecutively at flush (keeps LossDetector counters contiguous); Bulk-only, N=8/1ms. Honest scope: helps only the repair>0 (lossy) path — clean links already bypass the encoder. Both settle the "how do we reach WireGuard throughput" question: it's CPU/single- thread bound (FEC encode + one core), NOT I/O — so multi-core is the primary lever, batching compounds it on lossy links. Follow-ups filed as #27-#31. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two design specs answering "how do we reach WireGuard throughput?" The io_uring investigation already proved the answer isn't I/O — throughput is CPU/single-thread-bound (RaptorQ FEC encode ~24 µs/pkt + everything on one core). So:
2026-07-03-multicore-sharding-design.md(#10) — the primary leverPer-peer engine sharding: N independent single-thread
DataPlaneengines (one pinned per core), peers assigned by hash, each with its ownSO_REUSEPORTUDP socket + its ownIFF_MULTI_QUEUETUN queue. No shared state, no locks — preserves the Phase A per-flow latency win and reuses the existingrun_poll/run_uringloops verbatim (engines=1is byte-identical to today). Scales aggregate throughput across peers.Key honesty: yipd is a single connected peer today (
sock.connect), soSO_REUSEPORTneeds the multi-peer data plane (sub-project #2) — this ships the seams and is a deliberate no-op until then. Single-flow multi-Gbit (one fat link) needs the deferred crypto worker-pool (regime B, #28).2026-07-03-fec-object-batching-design.md— compounds it on lossy linksBatch N sealed packets into one RaptorQ object → amortize the ~24 µs
Encoder::newto ~24/N. Container framing + derived AEAD counters (no wire change); the accumulator buffers plaintext and seals consecutively at flush so theLossDetector's counters stay contiguous. Bulk-only, N=8 / 1 ms. Honest scope: clean links already bypass the encoder, so this helps only the repair>0 (lossy) path — not clean-link throughput.Follow-ups filed
#27 (batching impl), #28 (single-flow worker-pool, deferred), #29 (extend zero-repair bypass to non-Bulk), #30 (GSO for ARQ retx), #31 (RaptorQ encode SIMD/systematic). The specs' own follow-up sections capture smaller refinements (eBPF steering, session migration, NUMA, adaptive symbol_size, indexed RetxBuffer).
Design only — no code. Both want your sign-off on their open questions before implementation.
🤖 Generated with Claude Code