fork: revert hybrid-model checkpoint-creation gate (hydra_vortex #346, warmup hang on qwen35moe)#9
Closed
ddvnguyen wants to merge 1 commit into
Closed
Conversation
…n35moe, hydra_vortex ggml-org#346) PR #5 (dc8b569) added an is_rec term to the do_checkpoint gate so qwen35moe hybrid models get native context checkpoints on their own merits, not only as a side effect of --rpc-port. The new build (:6e397fb32, which combines PR #5 and PR #7) hangs in common_init_from_params' warmup llama_decode on the qwen35moe hybrid model with n_ctx_checkpoints=32 (the default), on both sm_120 (RTX) and sm_60 (P100). /health stays 503 'Loading model' indefinitely, the process is wedged in futex_do_wait with full GPU memory. Reverting to :f518cff16 (PR #7 only) recovers in ~30 s on RTX, ~3 min on P100. This reverts the gate change but keeps the other PR #5 work intact: the search path at line ~4031 still special-cases is_rec for recurrent/hybrid cache reuse, so hybrid models still benefit from the ik_llama.cpp#1762 port when --rpc-port is set. The hybrid gate will be reintroduced once the warmup hang is root-caused upstream (likely interaction between llama_decode warmup and checkpoint buffer pre-allocation for hybrid arch at large ctx). Tracks ddvnguyen/hydra_vortex#346. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ddvnguyen
pushed a commit
to ddvnguyen/hydra_vortex
that referenced
this pull request
Jun 24, 2026
… hybrid checkpoint gate) The previous OCI image (:6e397fb32, which combined fork PR #5 with PR #7) hangs in warmup llama_decode on the qwen35moe hybrid model with n_ctx_checkpoints=32 (the default), on both sm_120 (RTX) and sm_60 (P100). /health stays 503 'Loading model' indefinitely, the process is wedged in futex_do_wait with full GPU memory. The configs had already been rolled back to :f518cff16 to recover, but that loses PR #7's runtime-configurable StateGet chunk size. This bumps the OCI tags to :7fa82b18 (built from a 1-line revert of the is_rec term in the do_checkpoint gate that PR #5 added) and the submodule to 7fa82b1861fc9e3d14b5f5f88caa0f79a5d70e9b. PR #7 (StateGet chunk size) is kept intact; the search-path special-case from PR #5 is also kept, so hybrid/recurrent models still get cache reuse via the ik_llama.cpp#1762 port when --rpc-port is set. ### Cross-repo links - Hydra issue: #346 - Fork issue: ddvnguyen/llama.cpp#8 - Fork PR: ddvnguyen/llama.cpp#9 - Reverts fork PR #5 (dc8b56962), keeps fork PR #7 ### Submodule bump src/llama-cpp: 6e397fb32 (broken) -> 7fa82b18 (fixed) The pinned SHA is reachable on the fork's branch fork/hydra-346-revert-5-checkpoint-gate. ### OCI tags - ghcr.io/ddvnguyen/llama-server-sm60:7fa82b18 (md5 02c2791a...) - ghcr.io/ddvnguyen/llama-server-sm120:7fa82b18 (md5 c9b5f315...) ### Merge order The fork PR (#9) must be merged to ddvnguyen/llama.cpp:hydra-fork before this PR merges, so the submodule SHA becomes reachable on hydra-fork (the submodule's tracked branch). ### Verification plan - [ ] Fork PR #9 merged to hydra-fork - [ ] Merge this PR - [ ] bash scripts/deploy-hydra-head.sh all - [ ] curl http://localhost:8080/health -> 200 (RTX, ~30s after pull) - [ ] curl http://192.168.122.21:8086/health -> 200 (P100, ~3 min) - [ ] cold_concurrency E2E: prefill on RTX, save/restore to P100 Closes #346 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
6 tasks
ddvnguyen
pushed a commit
to ddvnguyen/hydra_vortex
that referenced
this pull request
Jun 24, 2026
…armup hang; PR #5 NOT the culprit) The previous attempt (revert of PR #5's is_rec term in the checkpoint gate, commit 7fa82b18) was the wrong bisect direction. Live verification on P100 shows :7fa82b18 still hangs in warmup at the exact same futex_do_wait state as :6e397fb32, with the same warmup log 'common_init_from_params: warming up the model ...'. Removing the checkpoint-creation gate term is not enough — the warmup hang is in PR #7, not PR #5. This reverts the parent-side config to :f518cff16 (PR #5 only, the last tag where model load returns within ~30s on RTX, ~3 min on P100). PR #7 (StateGet chunk size) is what's wedging the warmup; the revert was a 'slips to next milestone' per issue's suggested next steps. ### What - submodule: :7fa82b18 -> :f518cff16 (fork PR #5 only) - node-rtx.yaml: :7fa82b18 -> :f518cff16 - node-p100.yaml: :7fa82b18 -> :f518cff16 ### Cross-repo links - Hydra issue: #346 - Fork PR (closed, was wrong): ddvnguyen/llama.cpp#9 (reverted) - Bisect: f518cff16 works, 6e397fb32 hangs, 7fa82b18 hangs => PR #7 is the culprit ### Follow-up (out of scope for this PR) - The fork-side fix is to revert PR #7 in ddvnguyen/llama.cpp#7 and re-introduce it once the warmup hang is root-caused upstream (likely interaction between llama_decode warmup and the hydra_state_chunk_size field's 2 MiB default in llama_cparams). Filed as a separate issue once the root cause is known. - Hydra PR #338 (ConfigureStateChunkSizeAsync in WorkerSchedulerService.cs) becomes a no-op against the old engine; safe to keep on main. ### Verification plan - [ ] curl http://localhost:8080/health -> 200 (RTX, ~30s) - [ ] curl http://192.168.122.21:8086/health -> 200 (P100, ~3 min) - [ ] E2E: pytest tests/system -k cold_concurrency (P/D split) Closes #346 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements the C++ side of ddvnguyen/hydra_vortex#346.
What & why
PR #5 (dc8b569) added an
is_recterm to thedo_checkpointgate so qwen35moe hybrid models get native context checkpoints on their own merits, not only as a side effect of--rpc-port. The combined build(:6e397fb32, which has PR #5 + PR #7) hangs incommon_init_from_params' warmupllama_decodeon the qwen35moe hybrid model withn_ctx_checkpoints=32(the default), on both sm_120 (RTX) and sm_60 (P100)./healthstays 503 "Loading model" indefinitely; the process is wedged infutex_do_waitwith full GPU memory. Rolling back to:f518cff16(PR #5 only) recovers in ~30s on RTX, ~3 min on P100.This reverts the gate change but keeps the other PR #5 work intact: the search path at line ~4031 still special-cases
is_recfor recurrent/hybrid cache reuse, so hybrid models still benefit from the ik_llama.cpp#1762 port when--rpc-portis set (cross-node migration). The hybrid gate will be reintroduced once the warmup hang is root-caused upstream (suspected:llama_decodewarmup interacts badly with checkpoint-buffer pre-allocation for hybrid arch at large ctx).Cross-repo links
Files changed
tools/server/server-context.cpp: removed theis_recterm from thedo_checkpointgate (line 4193) and rewrote the comment block to document the revert + link to the follow-up issue.Test plan
cmake --build build_sm120_static --target llama-engine— OK (md5c9b5f3156980b98f3efed9b0d5575068)cmake --build build_sm60_static --target llama-engine— OK (md502c2791ae31c58af967a12dd0bc781c3)ghcr.io/ddvnguyen/llama-server-sm{60,120}:7fa82b18/healthflips to 200 (after parent PR merges and deploys)/healthflips to 200 (after parent PR merges and deploys)cold_concurrencyP/D split still works (prefill on RTX, save, restore to P100, decode)Submodule bump (parent side)
After this PR merges, the parent repo will bump the submodule to commit
7fa82b18in a follow-up PR withCloses #346.Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com