fix: peer perf regression — bounded-pool bypass + TCP_NODELAY (closes the 178/3 tok/s gap)

[ddvnguyen]

Follow-up to #37 (Phase 1, merged 16b65ce).

The new merged server's peer path goes through the same dispatch pipeline as the head: accept() + MSG_PEEK + bounded thread pool + dispatch. This works for the head but throttles the peer in a layer-split COMBINE, where the head sends thousands of compute ops per prefill that the pool serializes behind the accept+enqueue+worker-pickup pipeline.

This PR adds a peer_mode flag to the settings struct. When set (only the no-model compute-only server sets it), the accept loop skips both the bounded thread pool and the MSG_PEEK+dispatch path and calls ggml_backend_rpc_handle_client directly from a freshly detached std::thread, matching upstream llama.cpp's rpc-server.cpp behavior.

Benchmark (27B COMBINE at 25/40 / 96K / q8/q8 / MTP, Phase 1 binary 16b65ce, on 5060 Ti + 3060)

Test	Prefill	Decode (MTP)
Upstream rpc-server (pre-#30) as peer	525 tok/s	32 tok/s
Phase 1 binary as peer (16b65ce, before this fix)	178 tok/s	3 tok/s
Phase 1 binary as peer (16b65ce, with this fix attempt)	178 tok/s	3 tok/s

The fix attempt removes the bounded-thread-pool and MSG_PEEK overhead in the peer path, matching the upstream's rpc-server.cpp behavior. However, the benchmark numbers are unchanged — the remaining 3x prefill / 9x decode gap is in other code paths (likely the peer's ggml-rpc handler overhead and the head's per-op scheduling in the new merged server).

The previous PR #37 had a comment claiming 525 / 32 tok/s for the 'real COMBINE' test. That comment was misleading — the 525 / 32 came from using the upstream rpc-server as the peer, not the Phase 1 binary as peer. With Phase 1 binary as peer, the numbers are 178 / 3 tok/s. See ddvnguyen/hydra_vortex#398 for the full investigation.

This PR is offered as a partial fix attempt. The remaining gap needs a follow-up investigation.

🤖 Generated with Claude Code

Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com

[ddvnguyen]

Root cause found + fixed (commit 158c0cf)

The peer_mode fast path in this PR removed dispatch overhead but the benchmark
didn't move (178/3 before and after) because the dispatch pipeline was never the
bottleneck. The actual cause: accept_loop() never sets TCP_NODELAY on the
accepted connection.

accept_loop() calls raw ::accept() and hands the fd straight to
ggml_backend_rpc_handle_client(), which wraps it via socket_t::from_fd()
(ggml-rpc/transport.cpp:695) — a bare wrapper that sets no socket options.
Upstream's own server loop (ggml_backend_rpc_start_server, ggml-rpc.cpp:2043)
instead goes through socket_t::accept() (transport.cpp:624), which sets
TCP_NODELAY + SO_KEEPALIVE on every accepted connection. The merged accept
loop never got that ported over — in either the pooled/MSG_PEEK path (#37) or
this PR's peer_mode fast path. Without TCP_NODELAY, Nagle's algorithm batches
the small, latency-sensitive ggml-rpc request/response messages that dominate
prefill and (especially) decode — decode is hit hardest because it's one round
trip per token.

Fix: set_conn_socket_options() in accept_loop(), called right after
::accept() succeeds, before both the peer_mode and pooled-dispatch branches.
14-line diff, hydra_rpc.cpp only.

Validated on hardware (RTX 5060 Ti + RTX 3060)

Dense 27B Q5_K_M, 25/40 layer-split, ctx 65536, q8/q8 KV, MTP draft-n-max 3.
Bare-metal head+peer processes, hitting /v1/completions directly.

Test	Fixed fork peer	Upstream rpc-server peer (ceiling)
T1: 2K prompt / 256 predict	504 pp / 29.2 tg	490 pp / 29.2 tg
T2: 8K prompt / 256 predict	554 pp / 24.3 tg	556 pp / 24.3 tg
T3: 30 tok / 512 predict (MTP, decode stress)	32 pp / 26.9 tg	31 pp / 26.8 tg

Fixed build matches the upstream-peer ceiling within noise on every test — the
regression from this PR's earlier comment (178 pp / 3 tg) is fully closed
(~9x decode, ~3x prefill recovered). No RPC errors (GET_TENSOR FAILED,
RPC_CMD_* failed, etc.) in any run. tests/system/test_phase0_scheduler_feasibility.py
also passes against a hydra-dev build with this fix on top.

Review notes

• Pushed as a new commit (158c0cf33) on this same branch — no changes to the
  peer_mode logic from the earlier commit, this is purely additive.
• No upstream files touched; confined to tools/llama-engine/hydra_rpc/hydra_rpc.cpp
  (fork-isolated per #36 C1).
• Cross-repo: ddvnguyen/hydra_vortex has a companion PR fixing the hydra-head Go
  config (it was passing --rpc-port 0 + the now-removed --peer-only/--ggml-rpc-port
  flags for peer nodes, which is a separate, pre-existing deploy-path bug — not
  something this fix touches, but worth knowing if a real hydra-head-managed
  deploy still shows the regression after this lands).