fork: hydra_config inline T3 + hydra_metrics response (hydra_vortex #411)#48
Merged
Merged
Conversation
…ed semantics Phase 2b of #36 / ddvnguyen/hydra_vortex#397. Extends the existing 0x40 CONFIGURE handler (which previously handled only 'state_chunk_size') to accept a full common_params JSON delta classified by tier (T1/T2/T3). T1 keys apply immediately; T2/T3 keys are deferred to the next slot-free moment, when update_slots observes all slots idle. Implements the wire schema documented in ddvnguyen/hydra_vortex#406 (commit d06d9df specs/rpc-protocol.md). ## Tier model - T1 (apply immediately): sampling.*, n_predict, n_keep, seed, antiprompt, state_chunk_size. Sampler re-init + cparams field write; no rebuild. - T2 (defer; context rebuild): n_ctx, cache_type_k, cache_type_v, rope_*, yarn_*, attention_type. llama_free(ctx_tgt) + llama_new_context_with_model. - T3 (defer; model rebuild): n_gpu_layers, n_cpu_moe, override_tensor, split_mode, tensor_split, model.path, model.alias. load_model(new_params) (same call site as per-PREFILL model swap at server-context.cpp:3093). COMBINED-mode teardown (set_expert_mode=0, remove rpc backend, clear combined bindings) + reattach (preload rpc device for layer-split, rebind experts for expert-split) is wrapped around the model reload. ## Response shape (per ddvnguyen/hydra_vortex#406) success + tier + params_applied + deferred_keys + error. The legacy state_chunk_size_applied echo is kept (backward compat with the existing WorkerSchedulerService.cs:2842 startup call); new code reads params_applied. ## Drain timeout HYDRA_COORD_PROFILE_SWITCH_DRAIN_TIMEOUT (default 300s, capped to [1, 3600]) bounds how long the engine holds a pending config waiting for the slot-free trigger. On timeout, the pending config is discarded and a delayed response is queued (the next INFO call surfaces it; the operator can also poll via /v1/info once the engine supports it). ## Files - include/llama-hydra.h: 6 new mutators + pending_config accessor + drain timeout setter - src/llama-context.{h,cpp}: hydra_pending_config struct + hydra_pending_config_tier / set_at members on llama_context - src/llama-hydra.cpp: implementation of all 6 mutators - tools/server/server-task.{h,cpp}: extend server_task_result_hydra_engine with tier / params_applied / deferred_keys fields + to_json override - tools/server/server-context.cpp: rewrite the SERVER_TASK_TYPE_HYDRA_ENGINE_CONFIGURE case to classify + apply T1 + stage T2/T3; inject the drain trigger in update_slots (the existing slot-free short-circuit at server-context.cpp:3712-3728) - tests/test-hydra-configure-tier.cpp: 6-case unit test covering T1, T2, T3, failure, legacy backward-compat (no tier emitted), and non-CONFIGURE op isolation ## Known scope limits (documented in the design RFC at ddvnguyen/hydra_vortex#406's deferred-semantics section) - T3 model/context reload is staged via pending_config; the full impl-level model_tgt / ctx_tgt swap is the next iteration (the structural contract — T3 keys stored, apply triggered on slot-free, state cleared — is in place; the engine returns success on a T2/T3 request but logs a clear warning that the rebuild is best-effort until the model/context swap is wired). - T3 mutators are storage-only: set_override_tensor / set_split_mode record staged state in module statics; they do not actually re-place tensors (which would require the model reload above). - state_chunk_size_applied kept in the response for backward compat with the existing C# call site. ## Build status - sm_120 (RTX 5060 Ti): builds clean (10.4 MB binary) - sm_60 (Tesla P100): builds clean (10.4 MB binary; requires CMAKE_CUDA_HOST_COMPILER=/usr/bin/g++-14 since CUDA 12.9 does not support gcc 15+) - New test: test-hydra-configure-tier (6 cases, all pass) - Existing tests: test-hydra-state-chunk-size + test-hydra-checkpoint-policy still pass ## Cross-references - #36 (v4 design handoff, Phase 2) - #40 (fork-side Phase 2b issue) - ddvnguyen/hydra_vortex#397 (parent tracker) - ddvnguyen/hydra_vortex#402 (Phase 2a — EngineConfig + ModelRegistry; input to this work) - ddvnguyen/hydra_vortex#406 (parent-side docs PR — the wire schema this PR implements) Refs: #40, ddvnguyen/hydra_vortex#397, ddvnguyen/hydra_vortex#406 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…lot-free moment Phase 2b of #36 — follow-up to #41. The previous PR (#41) added the wire schema (tier classification, params_applied echo, deferred_keys list) but the actual T2/T3 apply step in the slot-free moment was a no-op (just cleared the staged state and invalidated the graph cache). This PR completes the work: the apply step now actually rebuilds llama_context (T2) or fully reloads llama_model (T3) when the slot-free moment fires. ## T2 apply (apply_t2_rebuild) - Reads the staged T2 keys (n_ctx, cache_type_k, cache_type_v, rope_*, yarn_*) from the pending_config JSON on the context - Updates params_base in-place - Frees the live llama_context (and ctx_dft if present) - Rebuilds llama_context_params from params_base - Recreates the context via llama_new_context_with_model - Re-inits per-slot samplers (the old samplers were bound to the freed context) - On failure: rollback to the old params_base and recreate. If the rollback itself fails, GGML_ABORT (the engine is in a bad state and must exit). ## T3 apply (apply_t3_rebuild) - Reads the staged T3 statics (n_gpu_layers, n_cpu_moe, model.path, split_mode, tensor_split, override_tensor) populated by the CONFIGURE handler - Builds swapped_params from params_base + staged statics - Parses the override_tensor wire string into llama_model_tensor_buft_override entries (via ggml_backend_dev_buffer_type() lookup; 'CPU' mapped to ggml_backend_cpu_buffer_type()) - COMBINED-mode teardown BEFORE the model reload: llama_hydra_set_expert_mode(0), hydra_remove_combined_rpc_backend(peer), llama_hydra_clear_combined_bindings(peer). The dual-loaded expert bindings must be cleared so the new ctx_tgt doesn't inherit stale peer device references. - Full model reload via the existing load_model() (same call site as the per-PREFILL model swap at server-context.cpp:3392). load_model() handles the unload of the current model, the load of the new model, the new context creation, MTP/draft paths, and slot rebuild. - COMBINED-mode reattach AFTER the reload: for layer-split, re-enable the mode flag (the new load_model already preloaded the peer device with the new tensor_split); for expert-split, re-resolve the peer's RPC device, rebind the expert tensors via llama_hydra_rebind_combined_experts(). Same fail-open pattern as SET_EXPERT_MODE — if the peer is unreachable, the engine stays solo. - On failure: rollback by reloading the old params_base. If the rollback itself fails, GGML_ABORT. ## Drain timeout - The apply step checks HYDRA_COORD_PROFILE_SWITCH_DRAIN_TIMEOUT (default 300s). On timeout, the staged config is discarded and the next INFO call surfaces the cleared state. ## Files - tools/server/server-context.cpp (+410/-8): adds apply_pending_hydra_config() (the high-level orchestrator), apply_t2_rebuild() (context-only rebuild), apply_t3_rebuild() (full model reload), and hydra_parse_cache_type() (helper for parsing wire-shape cache_type strings). The existing llama_hydra_apply_pending_config() low-level helper in llama-hydra.cpp remains in place (it now serves as the graph cache invalidator + state clearer; the slot-free check in update_slots now calls the new high-level method first, which does the actual rebuild before the low-level helper would clear the staged state). ## Build status - sm_120 (RTX 5060 Ti): builds clean - sm_60 (Tesla P100): builds clean (with CMAKE_CUDA_HOST_COMPILER=g++-14) - Tests: test-hydra-state-chunk-size, test-hydra-configure-tier, test-hydra-checkpoint-policy, test-hydra-rpc-bind — all pass ## Cross-references - #36 (v4 design handoff, Phase 2) - #40 (fork-side Phase 2b issue) - #41 (predecessor PR — wire schema + best-effort stub) - ddvnguyen/hydra_vortex#397 (parent tracker) - ddvnguyen/hydra_vortex#406 (parent-side docs PR — the wire schema) - ddvnguyen/hydra_vortex#407 (parent-side code PR — applier + overrides) - docs/system-test-phase-2b.md (live E2E test procedure) Refs: #41, ddvnguyen/hydra_vortex#406, ddvnguyen/hydra_vortex#407 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Addresses all review findings from the ddv-hydra review bot on #41 and #42. ## PR #42 P1: params_base stale after load_model() - Added comment in apply_t3_rebuild() confirming load_model() does params_base = params internally (line 844), so no explicit reassignment is needed after a successful load. ## PR #42 P1: T2 rollback log before GGML_ABORT - Added SRV_ERR log before the GGML_ABORT on rollback failure so the operator sees the unrecoverable-state warning in the log. - Added SRV_INF log on successful rollback so the operator can confirm the engine recovered to the old model. ## PR #42 P2: override_tensor pointer lifetime - Updated comment in apply_t3_rebuild() clarifying that the pattern C string in tensor_buft_overrides points into the staged static s_hydra_pending_override_tensor, which is valid for the entire load_model() call (load_model parses overrides synchronously before T3 statics are cleared). ## PR #41 P1: T3 statics thread model invariant - Added a hard-constraint comment block on the s_hydra_pending_* file-scope statics in llama-hydra.cpp explaining why they are safe without locking today (same-thread dispatch) and what must change if the thread model changes (move to llama_context or add mutex). ## PR #42 P2: hydra_peer vs hydra_current_peer verification - hydra_current_peer is the currently connected peer (may change on SET_EXPERT_MODE peer switch). hydra_peer is the configured peer (from startup YAML). The teardown removes the RPC backend for the CURRENT peer and clears bindings for the CONFIGURED peer. These can differ only during a peer-switch mid-reload, which is blocked by the same-thread dispatch invariant. The existing code is correct; the comment above the teardown block now explains this. ## PR #42 P3: dead drain timeout in llama-hydra.cpp stub - Not changed in this commit — the llama_hydra_apply_pending_config() stub is still called as a fallback for non-server callers. Left as-is for now; can be removed in a follow-up. Refs: #41 (review), #42 (review) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
#44) server_context::start_rpc_server() declared hydra_rpc_ctx as an automatic-storage local, then handed hydra_rpc::start() its address to retain in a process-lifetime singleton (hydra_rpc::state().hydra_ctx), read by every future RPC connection from a bounded-thread-pool worker thread. The local went out of scope the moment start_rpc_server() returned, so the singleton retained a dangling pointer into a freed stack frame — a stack-use-after-return. Whether a given request's queue_tasks/queue_results dereference through it "worked" depended entirely on whether the freed stack slot had been reused yet, which explains the intermittent nature of #43 (task processed, result queued, but the response body never reaching the client socket). Fix: give ctx static storage duration. start_rpc_server() only ever runs meaningfully once per process (hydra_rpc::start() itself guards double-start), so `static` gives it exactly the lifetime the singleton already assumes. Also make hydra_send_all/hydra_recv_all log on failure (fd, bytes transferred, errno) instead of silently returning false — every caller already treats a false return as "give up" but had no way to tell a real socket error from `-lv` output, which the issue itself flagged as a diagnostic gap. Verified live on hardware (RTX 5060 Ti, Qwen3.5-2B-Q8_0): INFO (0x41) and CONFIGURE (0x40) previously timed out after 10s against a raw socat client, a raw Python client, and C# RpcClient. All three now round-trip correctly, including a second request on the same persistent connection. test-hydra-rpc-bind, test-hydra-configure-tier, and dotnet Tests.Shared EngineOpcodeTests all still pass. Closes #43 Co-authored-by: Hydra Engineering <hydra-engineering@local> Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
- parse optional hydra_config from chat completion request body - compare model_path/split_mode/tensor_split against current engine state - first-load path: engine starts empty, loads model on first hydra_config - hydra_metrics emitted in every chat completion response - T3 rebuild handles empty-start (no ctx_tgt) gracefully - split_mode/tensor_split added to server_context_meta - llama_hydra_set_split_mode/override_tensor: allow staging without ctx Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- llama-engine.cpp: set_hydra_combined_static(true) when peer is registered for layer-split (was never called — SET_EXPERT_MODE combined always rejected as solo) - llama-engine.cpp: pass correct split_mode and peer_reachable to set_hydra_capabilities (was hardcoded to solo/false) - server-context.cpp: always populate hydra_metrics in the normal (model-loaded) code path, not just first-load path Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ddvnguyen
pushed a commit
to ddvnguyen/hydra_vortex
that referenced
this pull request
Jul 12, 2026
hydra-fork now includes the fork/hydra-config-hydra-metrics merge: - hydra_config inline T3 parsing - hydra_metrics in every response - T2/T3 apply path - combined_static activation fix - dangling hydra_rpc_ctx fix Fork PR: ddvnguyen/llama.cpp#48 Fork issue: ddvnguyen/llama.cpp#47 Hydra parent PR: #411 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This was referenced Jul 12, 2026
This was referenced Jul 12, 2026
ddvnguyen
pushed a commit
that referenced
this pull request
Jul 13, 2026
* fork: P0-1 head-bootstrap mode in llama-engine — reuse PR #48/#53 mechanism (hydra_vortex #49) When llama-engine starts with no --model but with --rpc-engine (indicating it's a head, not a peer), it now builds server_context + server_routes + Hydra RPC and waits for CONFIGURE(0x40) T3 to load the model. This reuses the existing first_load_pending / apply_pending_hydra_config() / apply_t3_rebuild() mechanism from PR #48/#53 — no parallel implementation. Key changes: - Add set_routes_ptr() public method to server_context (for head-bootstrap to wire routes_ptr without accessing impl directly) - Split no-model path in llama-engine.cpp: - !has_model && !has_peer: compute-only peer (unchanged) - !has_model && has_peer: head-bootstrap mode (new) - Head-bootstrap creates server_context + server_routes + Hydra RPC with empty backends, registers full HTTP inference routes - CONFIGURE T3 triggers apply_pending_hydra_config() via existing first_load_pending mechanism in update_slots() Verified has_peer signal: peer-only nodes (RTX 3060) never have --rpc-engine in their config, head-bootstrap nodes always do. * fork: P0-1 round 3 — fix is_ready, capabilities, null-meta guard (hydra_vortex #49) Three fixes for head-bootstrap mode: 1. ctx_http.is_ready now set immediately after start() — the server is ready from the moment HTTP starts, matching the compute-only-peer pattern. Individual routes handle the no-model-yet case themselves. 2. set_hydra_capabilities/set_hydra_combined_static called after deferred first-load via bootstrap_* fields staged at startup and applied in apply_pending_hydra_config() success path. Also registers local tensors, enables shared-backend compute lock, and updates RPC backends. 3. Null-meta guard in handle_completions_impl, post_chat_completions, and post_infill — returns 503 with clear message instead of crashing when meta is null (bootstrap window before first CONFIGURE). Also adds hydra_rpc::update_backends() for populating compute backends after deferred first-load. * fork: P0-1 complete null-meta guards across all meta-deref handlers (hydra_vortex #49) Added null-meta guards (returns 503 + clear message) to every handler that dereferences meta-> under meta_mutex: get_props, post_responses_oai, post_transcriptions_oai, post_anthropic_messages, post_anthropic_count_tokens, post_apply_template, get_models, get_model_info, post_rerank, handle_embeddings_impl Previously covered (round 3): handle_completions_impl, post_chat_completions, post_infill. The bootstrap window (is_ready=true, no model loaded yet) is now safe to expose to real traffic — all endpoints return a clear 503 instead of crashing on null deref. * fork: P0-1 add bootstrap_init() — required for head-bootstrap start_loop() (hydra_vortex #49) Without bootstrap_init(), the queue callbacks (on_new_task, on_update_slots, on_sleeping_state) are never wired up in head-bootstrap mode, causing std::bad_function_call crash when start_loop() tries to invoke them. bootstrap_init() wires up these callbacks + metrics.init() without requiring a model (no ctx_tgt/model_tgt assertions). Called before start_loop() in the head-bootstrap path. Verified via smoke test: - /health returns 200 {"status":"ok","mode":"bootstrap"} - /v1/chat/completions returns 501 "model not loaded — waiting for CONFIGURE" --------- Co-authored-by: Hydra Engineering <hydra-engineering@local>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements the C++ side of ddvnguyen/hydra_vortex#411.
What & why
C# coordinator now sends full model presets (hydra_config) on every request.
Engine parses it, compares against current state, and triggers inline T3 reload
if config differs. Every response includes hydra_metrics for observability.
Fork commits on this branch
ea74303b4— fix combined_static activation + always emit hydra_metricsb831c237c— hydra_config inline T3 + hydra_metrics responsec184c4fbd— fix dangling hydra_rpc_ctx stack pointer (fix: dangling hydra_rpc_ctx stack pointer breaks RPC response delivery #44)cac7122a3— review findings fixes on fork PRs fork: extend 0x40 CONFIGURE for T1/T2/T3 common_params delta + deferred semantics (Phase 2b) #41 and fork: implement T2/T3 apply path — actually reload model+context at slot-free moment #42dd521b105— T2/T3 apply path (reload model+context at slot-free moment)6830c2542— extend 0x40 CONFIGURE for T1/T2/T3 (Phase 2b)C++ changes
Test plan