fork: Phase 1 — merged Hydra RPC server in tools/llama-engine/hydra_rpc/#37
Conversation
Implements Phase 1 of #36 (the v4 design's "Cherry-pick + new module"). Extracts the unified server logic that PR #30 placed in tools/server/server-context.cpp into a fork-isolated module under tools/llama-engine/hydra_rpc/, and replaces the per-conn std::thread::detach() with a bounded_thread_pool<2> per #36's C7 hard constraint. What this does: - New fork-isolated module tools/llama-engine/hydra_rpc/ (~420 LOC): - bounded_thread_pool.h — template<2>, enqueue / try_enqueue / stop - hydra_rpc.h — public API: start(settings), stop(), is_running(), bound_port() - hydra_rpc.cpp — accept loop, MSG_PEEK dispatch (0x0E → ggml-rpc, else → hydra via the bridge trampoline), bounded pool, lifecycle - tools/server/server-context.cpp: server_context::start_rpc_server is now a thin adapter that builds hydra_rpc::settings and delegates to hydra_rpc::start(). start_rpc_accept_loop is removed (the new module owns the accept loop). The Hydra handler is reached through a new extern "C" trampoline hydra_rpc_bridge defined here. - tools/llama-engine/llama-engine.cpp: the no-model path that previously inlined the bind+listen+start_rpc_accept_loop now calls hydra_rpc::start() with hydra_ctx=nullptr (ggml-RPC only). - tools/server/server-rpc.h: doc-comment update pointing at the new module. The old start_rpc_accept_loop declaration is removed. - ggml/src/ggml-rpc/transport.cpp: cherry-picked from PR #31 — set_keepalive() on accept/connect + MSG_NOSIGNAL on send. Prevents idle-connection drops during 2+ minute model loads. Why this design (per #36): - C1: minimal upstream diff. The new module is fork-isolated, so rebase conflicts stay in tools/llama-engine/hydra_rpc/. The upstream-touched files are 4 (transport.cpp, llama-engine.cpp, server-context.cpp, server-rpc.h), but only transport.cpp has net-new functionality; the others are a refactor. - C7: bounded thread pool (size 2, max queue 64). The previous per-conn std::thread::detach() was unbounded — a connection flood could spawn unbounded threads. - Drop-on-overflow (try_enqueue, not enqueue) on the RPC server's accept path. A DoS-style flood sees drops, not back-pressure that fills the accept queue. Cherry-pick from #31: kept set_keepalive + MSG_NOSIGNAL (transport.cpp ~25 LOC). Discarded: RPC_STATUS_ASSERT → fail-soft replacement (C6 violation), response = {} zero-inits (C6), fattn.cu GGML_ABORT removal ("NOT cherry-picked" list), startup_stage + --peer-health-url + wait_for_peer_ready (Phase 1.5), llama_hydra_validate_quant_parity + llama_hydra_free_result (Phase C, split to ddvnguyen/hydra_vortex#394), and the --yarn-ext-ctx / --yarn-orig-ctx stripping (looks like a bug in #31, not propagated). Verification: - Build: cmake --build build_sm86_sm120 --target llama-engine — 21/21 targets, zero warnings introduced by the new module. - Phase 0 sanity test (tests/system/test_phase0_scheduler_feasibility.py, Qwen3.5-9B-Q8_0 50/50 layer-split across CUDA0 + RPC0): - Head /health 200 after ~70s - 22 prompt tokens, 32 completion tokens, 41.80 tok/s decode - graphs reused = 30 / 32 (scheduler placing compute on the right device across the RPC boundary) - 1 context checkpoint of 32 created - No GET_TENSOR FAILED, no RPC_CMD_* failed, no device-index errors, no truncated graph, no pre-allocated-tensor errors G3 (SOLO 5060 Ti ≥ 190 tok/s, no regression) deferred to hardware validation in the parent repo. G9 (upstream-touched ≤ 75 LOC across ≤ 4 functions in 2 files) — the upstream-touched is concentrated in transport.cpp (net new) and server-context.cpp (refactor extraction). The new module is fork-isolated. Cross-repo: - Hydra parent: ddvnguyen/hydra_vortex#398 (the per-task tracker) - Fork issue: #36 (the v4 design handoff) - Branch: fork/hydra-392-phase-1-merged-rpc Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Phase 1 — Live Test Results (base 16b65ce, build 9563)Live-tested the Phase 1 fork binary on the 5060 Ti + 3060 stack with the 27B DENSE Q5_K_M (the production's G1 model), MTP speculative, 96K context, q8_0/q8_0 KV cache, layer-split 25/40 (production's config). All builds in this run use TL;DR
Test matrix
Finding 1 (CRITICAL):
|
| Metric | Fork (Phase 1, 9563) | Upstream (4a7ee31) | Delta |
|---|---|---|---|
| PREFILL (2969 tok, max_tokens=1) | 422.91 tok/s | 286.05 tok/s | fork +48% |
| DECODE (29+512 tok, MTP) | 24.21 tok/s | 21.46 tok/s | fork +13% |
| MTP draft acceptance | 90.7% | 90.9% | identical |
| Model split | 11.06 GB CUDA0 + 6.46 GB RPC0 | identical | identical |
| Wall-clock prefill | 7.0s | 10.4s | fork -32% |
| VRAM 5060 Ti | 14699 MiB | 14699 MiB | identical |
The fork's speedup is from the Phase 1 work (PR #30 + this PR's #37):
hydra: registered 866 resident tensor(s) for zero-copy RPC resolution (epoch bumped)(fork-specific)hydra_rpc: unified server on 0.0.0.0:18081(single accept loop, bounded thread pool, drop-on-overflow)set_keepalive+MSG_NOSIGNALon RPC sockets (cherry-picked from fork: resolve #376 — make COMBINE mode RPC work (gate-then-degrade) #31)
The architecture change (one process with merged server vs two processes with separate accept loops) is not the dominant factor — fat upstream with FA matches separate upstream with FA to within 1-2% on the same model/config.
Finding 3: G1 not met on this hardware pair
#36 set G1 = "DENSE 27B layer-split COMBINED decode ≥ 50 tok/s". The 27B on the 5060 Ti + 3060 with 25/40 split gets 19.19–24.70 tok/s decode. The 3060 is the bottleneck — 38% of the 27B's compute on a 3-4× slower GPU. Hitting 50 tok/s on this exact pair isn't possible without:
- A different layer-split ratio (but the 3060 can't fit 50% of a 19 GB model in 12 GB)
- A different peer (P100, or another 5060 Ti)
- The P/D split path (prefill 5060 Ti, decode layer-split) — not implemented yet, this is Phase 4-5 of llama-engine: per-GPU llama-engines + merged Hydra RPC + no-weight-transfer COMBINE #36
The Phase 1 architecture is correct; the G1 number as written is aspirational for the current hardware pair.
Open follow-ups
-DGGML_CUDA_FA_ALL_QUANTS=OFF→ONinDevelopmentRunBook.md. (See finding 1.) Separate PR, needs an OCI rebuild.- Production
--yarn-ext-ctx 131072config error ininfra/hydra-head/config/node-rtx-27b.yaml:40— the binary doesn't accept this flag. Pre-existing config drift. The model metadata says native 262K, so the YaRN config (yarn-orig-ctx 32K, rope-scale 4) is also stale. - Spike 381 build commands missing
-DGGML_CUDA_FA=ON— both sm_86 and sm_120 builds need it. Without it, the bare-spec prefill is 151 tok/s (off by 2x). Should be added to spike 381 and runbook. - Production binary on the container is still the pre-fork: Unified RPC server — one port, one binary, no extra flags (closes #29) #30 build (
9561). To deploy Phase 1, runbash scripts/deploy-hydra-head.sh rtxto rebuildhydra-head:rtxwith9563. - Unit tests for Phase 1 (
test-hydra-rpc-accept-loop,test-hydra-bounded-thread-pool,test-hydra-solo-regression) are still TODO. - Parent submodule bump PR in
ddvnguyen/hydra_vortex— the local submodule in the working tree is at16b65ce7, but the parent's tracked pointer is still97ed5566(pre-fork: Unified RPC server — one port, one binary, no extra flags (closes #29) #30). Per the workflow, this PR opens after the fork PR is merged (it is). The local bump was rolled back per the "Do not merge" instruction.
Cross-repo
- Fork PR:
ddvnguyen/llama.cpp#37(this PR, 16b65ce) - Design:
ddvnguyen/llama.cpp#36 - Parent epic:
ddvnguyen/hydra_vortex#392 - Per-task tracker:
ddvnguyen/hydra_vortex#398 - Spike 381 (related measurement plan):
ddvnguyen/hydra_vortex#381
What I want from the senior reviewer
- Sanity check the FA_ALL_QUANTS=OFF finding — does this match your understanding of the FA backend? Is the recommended fix correct, or is there a different one (e.g., fix the non-FA fallback instead)?
- Approve changing the runbook flag to
FA_ALL_QUANTS=ON(and the corresponding OCI rebuild)? - G1 hardware implication — is the 50 tok/s target worth re-evaluating given the 3060's compute ceiling, or accept the 24 tok/s as the realistic ceiling for this pair?
- Unit tests — add in this PR or a follow-up?
- Should the misdiagnosed "sm_86 FA cap" be retracted from convert-pth-to-ggml.py error with "Got unsupported ScalarType BFloat16" ggml-org/llama.cpp#398 and the earlier comment on this PR? Or kept with a "partially wrong" note?
Phase 1 — Live Test Results (base 16b65ce, build 9563)Live-tested the Phase 1 fork binary on the 5060 Ti + 3060 stack with the 27B DENSE Q5_K_M (the production's G1 model), MTP speculative, 96K context, q8_0/q8_0 KV cache, layer-split 25/40 (production's config). All builds in this run use TL;DR
Test matrix
Finding 1 (CRITICAL):
|
| Metric | Fork (Phase 1, 9563) | Upstream (4a7ee31) | Delta |
|---|---|---|---|
| PREFILL (2969 tok, max_tokens=1) | 422.91 tok/s | 286.05 tok/s | fork +48% |
| DECODE (29+512 tok, MTP) | 24.21 tok/s | 21.46 tok/s | fork +13% |
| MTP draft acceptance | 90.7% | 90.9% | identical |
| Model split | 11.06 GB CUDA0 + 6.46 GB RPC0 | identical | identical |
| Wall-clock prefill | 7.0s | 10.4s | fork -32% |
| VRAM 5060 Ti | 14699 MiB | 14699 MiB | identical |
The fork's speedup is from the Phase 1 work (PR #30 + this PR's #37):
hydra: registered 866 resident tensor(s) for zero-copy RPC resolution (epoch bumped)(fork-specific)hydra_rpc: unified server on 0.0.0.0:18081(single accept loop, bounded thread pool, drop-on-overflow)set_keepalive+MSG_NOSIGNALon RPC sockets (cherry-picked from fork: resolve #376 — make COMBINE mode RPC work (gate-then-degrade) #31)
The architecture change (one process with merged server vs two processes with separate accept loops) is not the dominant factor — fat upstream with FA matches separate upstream with FA to within 1-2% on the same model/config.
Finding 3: G1 not met on this hardware pair
#36 set G1 = "DENSE 27B layer-split COMBINED decode ≥ 50 tok/s". The 27B on the 5060 Ti + 3060 with 25/40 split gets 19.19–24.70 tok/s decode. The 3060 is the bottleneck — 38% of the 27B's compute on a 3-4× slower GPU. Hitting 50 tok/s on this exact pair isn't possible without:
- A different layer-split ratio (but the 3060 can't fit 50% of a 19 GB model in 12 GB)
- A different peer (P100, or another 5060 Ti)
- The P/D split path (prefill 5060 Ti, decode layer-split) — not implemented yet, this is Phase 4-5 of llama-engine: per-GPU llama-engines + merged Hydra RPC + no-weight-transfer COMBINE #36
The Phase 1 architecture is correct; the G1 number as written is aspirational for the current hardware pair.
Open follow-ups
-DGGML_CUDA_FA_ALL_QUANTS=OFF→ONinDevelopmentRunBook.md. (See finding 1.) Separate PR, needs an OCI rebuild.- Production
--yarn-ext-ctx 131072config error ininfra/hydra-head/config/node-rtx-27b.yaml:40— the binary doesn't accept this flag. Pre-existing config drift. The model metadata says native 262K, so the YaRN config (yarn-orig-ctx 32K, rope-scale 4) is also stale. - Spike 381 build commands missing
-DGGML_CUDA_FA=ON— both sm_86 and sm_120 builds need it. Without it, the bare-spec prefill is 151 tok/s (off by 2x). Should be added to spike 381 and runbook. - Production binary on the container is still the pre-fork: Unified RPC server — one port, one binary, no extra flags (closes #29) #30 build (
9561). To deploy Phase 1, runbash scripts/deploy-hydra-head.sh rtxto rebuildhydra-head:rtxwith9563. - Unit tests for Phase 1 (
test-hydra-rpc-accept-loop,test-hydra-bounded-thread-pool,test-hydra-solo-regression) are still TODO. - Parent submodule bump PR in
ddvnguyen/hydra_vortex— the local submodule in the working tree is at16b65ce7, but the parent's tracked pointer is still97ed5566(pre-fork: Unified RPC server — one port, one binary, no extra flags (closes #29) #30). Per the workflow, this PR opens after the fork PR is merged (it is). The local bump was rolled back per the "Do not merge" instruction.
Cross-repo
- Fork PR:
ddvnguyen/llama.cpp#37(this PR, 16b65ce) - Design:
ddvnguyen/llama.cpp#36 - Parent epic:
ddvnguyen/hydra_vortex#392 - Per-task tracker:
ddvnguyen/hydra_vortex#398 - Spike 381 (related measurement plan):
ddvnguyen/hydra_vortex#381
What I want from the senior reviewer
- Sanity check the FA_ALL_QUANTS=OFF finding — does this match your understanding of the FA backend? Is the recommended fix correct, or is there a different one (e.g., fix the non-FA fallback instead)?
- Approve changing the runbook flag to
FA_ALL_QUANTS=ON(and the corresponding OCI rebuild)? - G1 hardware implication — is the 50 tok/s target worth re-evaluating given the 3060's compute ceiling, or accept the 24 tok/s as the realistic ceiling for this pair?
- Unit tests — add in this PR or a follow-up?
- Should the misdiagnosed "sm_86 FA cap" be retracted from convert-pth-to-ggml.py error with "Got unsupported ScalarType BFloat16" ggml-org/llama.cpp#398 and the earlier comment on this PR? Or kept with a "partially wrong" note?
1 similar comment
Phase 1 — Live Test Results (base 16b65ce, build 9563)Live-tested the Phase 1 fork binary on the 5060 Ti + 3060 stack with the 27B DENSE Q5_K_M (the production's G1 model), MTP speculative, 96K context, q8_0/q8_0 KV cache, layer-split 25/40 (production's config). All builds in this run use TL;DR
Test matrix
Finding 1 (CRITICAL):
|
| Metric | Fork (Phase 1, 9563) | Upstream (4a7ee31) | Delta |
|---|---|---|---|
| PREFILL (2969 tok, max_tokens=1) | 422.91 tok/s | 286.05 tok/s | fork +48% |
| DECODE (29+512 tok, MTP) | 24.21 tok/s | 21.46 tok/s | fork +13% |
| MTP draft acceptance | 90.7% | 90.9% | identical |
| Model split | 11.06 GB CUDA0 + 6.46 GB RPC0 | identical | identical |
| Wall-clock prefill | 7.0s | 10.4s | fork -32% |
| VRAM 5060 Ti | 14699 MiB | 14699 MiB | identical |
The fork's speedup is from the Phase 1 work (PR #30 + this PR's #37):
hydra: registered 866 resident tensor(s) for zero-copy RPC resolution (epoch bumped)(fork-specific)hydra_rpc: unified server on 0.0.0.0:18081(single accept loop, bounded thread pool, drop-on-overflow)set_keepalive+MSG_NOSIGNALon RPC sockets (cherry-picked from fork: resolve #376 — make COMBINE mode RPC work (gate-then-degrade) #31)
The architecture change (one process with merged server vs two processes with separate accept loops) is not the dominant factor — fat upstream with FA matches separate upstream with FA to within 1-2% on the same model/config.
Finding 3: G1 not met on this hardware pair
#36 set G1 = "DENSE 27B layer-split COMBINED decode ≥ 50 tok/s". The 27B on the 5060 Ti + 3060 with 25/40 split gets 19.19–24.70 tok/s decode. The 3060 is the bottleneck — 38% of the 27B's compute on a 3-4× slower GPU. Hitting 50 tok/s on this exact pair isn't possible without:
- A different layer-split ratio (but the 3060 can't fit 50% of a 19 GB model in 12 GB)
- A different peer (P100, or another 5060 Ti)
- The P/D split path (prefill 5060 Ti, decode layer-split) — not implemented yet, this is Phase 4-5 of llama-engine: per-GPU llama-engines + merged Hydra RPC + no-weight-transfer COMBINE #36
The Phase 1 architecture is correct; the G1 number as written is aspirational for the current hardware pair.
Open follow-ups
-DGGML_CUDA_FA_ALL_QUANTS=OFF→ONinDevelopmentRunBook.md. (See finding 1.) Separate PR, needs an OCI rebuild.- Production
--yarn-ext-ctx 131072config error ininfra/hydra-head/config/node-rtx-27b.yaml:40— the binary doesn't accept this flag. Pre-existing config drift. The model metadata says native 262K, so the YaRN config (yarn-orig-ctx 32K, rope-scale 4) is also stale. - Spike 381 build commands missing
-DGGML_CUDA_FA=ON— both sm_86 and sm_120 builds need it. Without it, the bare-spec prefill is 151 tok/s (off by 2x). Should be added to spike 381 and runbook. - Production binary on the container is still the pre-fork: Unified RPC server — one port, one binary, no extra flags (closes #29) #30 build (
9561). To deploy Phase 1, runbash scripts/deploy-hydra-head.sh rtxto rebuildhydra-head:rtxwith9563. - Unit tests for Phase 1 (
test-hydra-rpc-accept-loop,test-hydra-bounded-thread-pool,test-hydra-solo-regression) are still TODO. - Parent submodule bump PR in
ddvnguyen/hydra_vortex— the local submodule in the working tree is at16b65ce7, but the parent's tracked pointer is still97ed5566(pre-fork: Unified RPC server — one port, one binary, no extra flags (closes #29) #30). Per the workflow, this PR opens after the fork PR is merged (it is). The local bump was rolled back per the "Do not merge" instruction.
Cross-repo
- Fork PR:
ddvnguyen/llama.cpp#37(this PR, 16b65ce) - Design:
ddvnguyen/llama.cpp#36 - Parent epic:
ddvnguyen/hydra_vortex#392 - Per-task tracker:
ddvnguyen/hydra_vortex#398 - Spike 381 (related measurement plan):
ddvnguyen/hydra_vortex#381
What I want from the senior reviewer
- Sanity check the FA_ALL_QUANTS=OFF finding — does this match your understanding of the FA backend? Is the recommended fix correct, or is there a different one (e.g., fix the non-FA fallback instead)?
- Approve changing the runbook flag to
FA_ALL_QUANTS=ON(and the corresponding OCI rebuild)? - G1 hardware implication — is the 50 tok/s target worth re-evaluating given the 3060's compute ceiling, or accept the 24 tok/s as the realistic ceiling for this pair?
- Unit tests — add in this PR or a follow-up?
- Should the misdiagnosed "sm_86 FA cap" be retracted from convert-pth-to-ggml.py error with "Got unsupported ScalarType BFloat16" ggml-org/llama.cpp#398 and the earlier comment on this PR? Or kept with a "partially wrong" note?
Correction: the previous "Live Test Results" comments used the wrong peerThe 525 tok/s prefill / 32 tok/s decode numbers from the previous comments on this PR were from running the upstream
The 3× prefill / 9× decode regression is real. The new merged server's peer path has additional dispatch overhead compared to the upstream's tl;dr: this PR (#37) IS correct for the model-loaded head path with MTP, but the compute-only peer path is slower than the upstream's |
hydra_rpc.cpp's accept_loop() calls raw ::accept() and hands the fd straight to ggml_backend_rpc_handle_client(), which wraps it via socket_t::from_fd() (ggml-rpc/transport.cpp) — a bare wrapper that sets no socket options. Upstream's own server loop goes through socket_t::accept() instead, which sets TCP_NODELAY + SO_KEEPALIVE on every accepted connection. The merged accept loop never got that ported over, in either the pooled/MSG_PEEK path (#37) or this branch's peer_mode fast path. Without TCP_NODELAY, Nagle's algorithm batches the small, latency-sensitive ggml-rpc request/response messages that dominate prefill and (especially) decode — this is why the peer_mode change alone didn't move the earlier 178/3 tok/s benchmark at all. Validated on RTX 5060 Ti + RTX 3060 (Dense 27B Q5_K_M, 25/40 layer-split, ctx 65536, q8/q8 KV, MTP draft-n-max 3), bare-metal head+peer processes talking directly to /v1/completions: | Test | Fixed fork peer | Upstream rpc-server peer (ceiling) | |-----------------------------|-----------------|-------------------------------------| | 2K prompt / 256 predict | 504 pp / 29.2 tg | 490 pp / 29.2 tg | | 8K prompt / 256 predict | 554 pp / 24.3 tg | 556 pp / 24.3 tg | | 30 tok / 512 predict (MTP) | 32 pp / 26.9 tg | 31 pp / 26.8 tg | Fixed build matches the upstream-peer ceiling within noise on every test (previous regression was 178 pp / 3 tg). No RPC errors (GET_TENSOR FAILED etc.) in any run; test_phase0_scheduler_feasibility.py passes against a hydra-dev build with this fix. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
… the 178/3 tok/s gap) (#39) * fix: peer in compute-only mode skips bounded thread pool + MSG_PEEK The new merged server's peer path (started by PR #37) goes through the same dispatch pipeline as the head: accept() + MSG_PEEK + bounded thread pool + dispatch. This works for the head (where the pool bounds public-internet DoS risk) but throttles the peer in a layer-split COMBINE, where the head sends thousands of compute ops per prefill that the pool serializes behind the accept+enqueue+worker-pickup pipeline. This commit adds a peer_mode flag to the settings struct. When set (only the no-model compute-only server sets it), the accept loop skips both the bounded thread pool and the MSG_PEEK+dispatch path and calls ggml_backend_rpc_handle_client directly from a freshly detached std::thread, matching upstream llama.cpp's rpc-server.cpp behavior byte-for-byte. This restores the head's compute graph throughput on the peer path. Note: live benchmark on the 27B COMBINE at 25/40 / 96K / q8/q8 / MTP still showed 178 tok/s prefill vs 525 tok/s with the upstream rpc-server (3x slower). The remaining gap is in other code paths beyond the dispatch pipeline (likely the peer's ggml-rpc handler overhead and the head's per-op scheduling). A follow-up investigation is needed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: set TCP_NODELAY/SO_KEEPALIVE on merged-server accept loop hydra_rpc.cpp's accept_loop() calls raw ::accept() and hands the fd straight to ggml_backend_rpc_handle_client(), which wraps it via socket_t::from_fd() (ggml-rpc/transport.cpp) — a bare wrapper that sets no socket options. Upstream's own server loop goes through socket_t::accept() instead, which sets TCP_NODELAY + SO_KEEPALIVE on every accepted connection. The merged accept loop never got that ported over, in either the pooled/MSG_PEEK path (#37) or this branch's peer_mode fast path. Without TCP_NODELAY, Nagle's algorithm batches the small, latency-sensitive ggml-rpc request/response messages that dominate prefill and (especially) decode — this is why the peer_mode change alone didn't move the earlier 178/3 tok/s benchmark at all. Validated on RTX 5060 Ti + RTX 3060 (Dense 27B Q5_K_M, 25/40 layer-split, ctx 65536, q8/q8 KV, MTP draft-n-max 3), bare-metal head+peer processes talking directly to /v1/completions: | Test | Fixed fork peer | Upstream rpc-server peer (ceiling) | |-----------------------------|-----------------|-------------------------------------| | 2K prompt / 256 predict | 504 pp / 29.2 tg | 490 pp / 29.2 tg | | 8K prompt / 256 predict | 554 pp / 24.3 tg | 556 pp / 24.3 tg | | 30 tok / 512 predict (MTP) | 32 pp / 26.9 tg | 31 pp / 26.8 tg | Fixed build matches the upstream-peer ceiling within noise on every test (previous regression was 178 pp / 3 tg). No RPC errors (GET_TENSOR FAILED etc.) in any run; test_phase0_scheduler_feasibility.py passes against a hydra-dev build with this fix. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Hydra Engineering <hydra-engineering@local> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Summary
Implements Phase 1 of
ddvnguyen/llama.cpp#36— the v4 design's "Cherry-pick + new module". Extracts the unified server logic thatddvnguyen/llama.cpp#30placed intools/server/server-context.cppinto a fork-isolated module undertools/llama-engine/hydra_rpc/, and replaces the per-connstd::thread::detach()with abounded_thread_pool<2>per #36's C7 hard constraint.What this does
tools/llama-engine/hydra_rpc/(~420 LOC):bounded_thread_pool.h—template<2>,enqueue/try_enqueue/stophydra_rpc.h— public API:start(settings),stop(),is_running(),bound_port()hydra_rpc.cpp— accept loop, MSG_PEEK dispatch (0x0E→ ggml-rpc, else → hydra via the bridge trampoline), bounded pool, lifecycletools/server/server-context.cpp:server_context::start_rpc_serveris now a thin adapter that buildshydra_rpc::settingsand delegates tohydra_rpc::start().start_rpc_accept_loopis removed (the new module owns the accept loop). The Hydra handler is reached through a newextern "C"trampolinehydra_rpc_bridgedefined here.tools/llama-engine/llama-engine.cpp: the no-model path that previously inlined the bind+listen+start_rpc_accept_loopnow callshydra_rpc::start()withhydra_ctx=nullptr(ggml-RPC only).tools/server/server-rpc.h: doc-comment update pointing at the new module. The oldstart_rpc_accept_loopdeclaration is removed.ggml/src/ggml-rpc/transport.cpp: cherry-picked from PR fork: resolve #376 — make COMBINE mode RPC work (gate-then-degrade) #31 —set_keepalive()on accept/connect +MSG_NOSIGNALon send. Prevents idle-connection drops during 2+ minute model loads.Why this design (per #36)
tools/llama-engine/hydra_rpc/. The upstream-touched files are 4 (transport.cpp, llama-engine.cpp, server-context.cpp, server-rpc.h), but onlytransport.cpphas net-new functionality; the others are a refactor.std::thread::detach()was unbounded — a connection flood could spawn unbounded threads.try_enqueue, notenqueue) on the RPC server's accept path. A DoS-style flood sees drops, not back-pressure that fills the accept queue.Cherry-pick from #31
Kept (transport.cpp, ~25 LOC):
set_keepaliveon accept/connectMSG_NOSIGNALonsendDiscarded (per #36 C6 and "NOT cherry-picked" list):
RPC_STATUS_ASSERT→ fail-soft replacement in buffer ops — C6 violation (only keep null-guards for teardown; normal ops stay fail-stop)response = {}zero-inits (3 places) — revertggml-cuda/fattn.cuGGML_ABORTremoval — per design, "Do NOT cherry-pick from PR fix: COMBINED layer-split startup crash + performance improvements #34"startup_stagestate machine +--peer-health-url+wait_for_peer_ready— Phase 1.5, deferllama_hydra_validate_quant_parity+llama_hydra_free_result— Phase C, split toddvnguyen/hydra_vortex#394--yarn-ext-ctx/--yarn-orig-ctxstripping in the filter — looks like a bug in fork: resolve #376 — make COMBINE mode RPC work (gate-then-degrade) #31, not propagatedVerification
cmake --build build_sm86_sm120 --target llama-engine— 21/21 targets, zero warnings introduced by the new module.tests/system/test_phase0_scheduler_feasibility.pyin the parent repo, Qwen3.5-9B-Q8_0 50/50 layer-split across CUDA0 + RPC0):/health200 after ~70sgraphs reused = 30 / 32(scheduler placing compute on the right device across the RPC boundary)GET_TENSOR FAILED, noRPC_CMD_* failed, nodevice-indexerrors, notruncated graph, nopre-allocated-tensorerrorsG3 (SOLO 5060 Ti ≥ 190 tok/s, no regression) deferred to hardware validation in the parent repo. G9 (upstream-touched ≤ 75 LOC across ≤ 4 functions in 2 files) — the upstream-touched is concentrated in
transport.cpp(net new) andserver-context.cpp(refactor extraction). The new module is fork-isolated.Open-question resolutions
set_hydra_capabilitiescall site.Cross-repo
ddvnguyen/hydra_vortex#398(the per-task tracker, milestone "Llama-Engine - P/D split mix-quant")ddvnguyen/llama.cpp#36(the v4 design handoff)ddvnguyen/fork/hydra-392-phase-1-merged-rpc(this PR)Test plan
cmake --build build_sm86_sm120 --target llama-engine— cleancmake --build build_sm60 --target llama-engine— pending (not on the build host for this commit)tests/system/test_phase0_scheduler_feasibility.py)test-hydra-rpc-accept-loop,test-hydra-bounded-thread-pool,test-hydra-solo-regression— to be added in a follow-up commit (seeddvnguyen/hydra_vortex#398)