feat(cluster): add TLS 1.3 to replica plane with PSK channel binding#3461
Open
hubcio wants to merge 1 commit into
Open
feat(cluster): add TLS 1.3 to replica plane with PSK channel binding#3461hubcio wants to merge 1 commit into
hubcio wants to merge 1 commit into
Conversation
The replica port carried plaintext: the PSK handshake authenticates cluster membership but gives no confidentiality, and encryption was delegated to out-of-band tunnels (WireGuard/VPC). Replica traffic now supports in-process TLS behind an opt-in [cluster.tls] table. TLS session state cannot cross shard boundaries, so shard 0 no longer runs the PSK handshake before delegating: it blindly ships the raw fd right after accept or dial, and the owning shard wraps it and runs the handshake inside the TLS stream, both under one handshake_grace budget. With handshakes no longer serialized on shard 0, unauthenticated inflight connections get a global cap: a shard-0 slot table, an outcome ack from the owning shard, and a deadline that reclaims slots when an ack is lost. The same ack clears a pending-dial set that stops the periodic reconnect sweep from double-dialing a peer whose handshake is still in flight. When the owning shard is shard 0 itself the ack releases inline on the local bus instead of self-sending a frame. TLS 1.3 only, ALPN "iggy-replica". Two cert modes mirror the legacy TcpTlsConfig shape: CA files (default; the new ca_file anchors the dialer, a TLS-client role the server plane never has) and self_signed (auto-generated cert, accept-any verifier). Both modes require cluster.auth: neither carries client certificates, so TLS authenticates the acceptor only and the PSK handshake stays the sole peer authenticator. The PSK MAC absorbs the TLS exporter value plus a mode byte: a relay MITM terminating both legs produces two different exporters and the handshake fails. The exporter is only reachable by handshaking with futures-rustls directly and converting into the compio-tls stream via its public From impls; a tripwire test pins that route. The transcript change is wire-incompatible with the previous MAC, acceptable while server-ng is unreleased. Also fixes framing::write_message never flushing: rustls holds ciphertext in its internal buffer until flushed, so handshake frames written over TLS never reached the wire (no-op on plain TCP).
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #3461 +/- ##
=============================================
- Coverage 74.56% 60.39% -14.18%
Complexity 937 937
=============================================
Files 1249 1248 -1
Lines 123564 112691 -10873
Branches 99839 88997 -10842
=============================================
- Hits 92133 68057 -24076
- Misses 28448 41540 +13092
- Partials 2983 3094 +111
🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The replica port carried plaintext: the PSK handshake authenticates
cluster membership but gives no confidentiality, and encryption was
delegated to out-of-band tunnels (WireGuard/VPC). Replica traffic
now supports in-process TLS behind an opt-in [cluster.tls] table.
TLS session state cannot cross shard boundaries, so shard 0 no
longer runs the PSK handshake before delegating: it blindly ships
the raw fd right after accept or dial, and the owning shard wraps
it and runs the handshake inside the TLS stream, both under one
handshake_grace budget. With handshakes no longer serialized on
shard 0, unauthenticated inflight connections get a global cap: a
shard-0 slot table, an outcome ack from the owning shard, and a
deadline that reclaims slots when an ack is lost. The same ack
clears a pending-dial set that stops the periodic reconnect sweep
from double-dialing a peer whose handshake is still in flight.
When the owning shard is shard 0 itself the ack releases inline
on the local bus instead of self-sending a frame.
TLS 1.3 only, ALPN "iggy-replica". Two cert modes mirror the
legacy TcpTlsConfig shape: CA files (default; the new ca_file
anchors the dialer, a TLS-client role the server plane never has)
and self_signed (auto-generated cert, accept-any verifier). Both
modes require cluster.auth: neither carries client certificates,
so TLS authenticates the acceptor only and the PSK handshake stays
the sole peer authenticator. The PSK MAC absorbs the TLS exporter
value plus a mode byte: a relay MITM terminating both legs
produces two different exporters and the handshake fails. The
exporter is only reachable by handshaking with futures-rustls
directly and converting into the compio-tls stream via its public
From impls; a tripwire test pins that route. The transcript change
is wire-incompatible with the previous MAC, acceptable while
server-ng is unreleased.
Also fixes framing::write_message never flushing: rustls holds
ciphertext in its internal buffer until flushed, so handshake
frames written over TLS never reached the wire (no-op on plain
TCP).