Deduplicate q4 quantization functions#383
Merged
Merged
Conversation
f59fe8e to
e0fe526
Compare
Contributor
Author
|
I've added a basic test for the quantization functions, but it is failing on macOS due to an illegal instruction. I suspect that the CMakeLists.txt selects some optimization flags that the CI machine does not support. I've disabled the test on macOS for now as I don't have macOS and don't want to hammer this PR further with pushes. |
Member
Weird, the test passes on my M1. |
ggerganov
approved these changes
Mar 22, 2023
Seunghhon
pushed a commit
to Seunghhon/llama.cpp
that referenced
this pull request
Apr 26, 2026
* Deduplicate q4 quantization functions * Use const; add basic test * Re-enable quantization test * Disable AVX2 flags in CI --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
phuongncn
pushed a commit
to phuongncn/llama.cpp-gx10-dgx-sparks-deepseekv4
that referenced
this pull request
Apr 28, 2026
* Deduplicate q4 quantization functions * Use const; add basic test * Re-enable quantization test * Disable AVX2 flags in CI --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
ljubomirj
pushed a commit
to ljubomirj/llama.cpp
that referenced
this pull request
May 6, 2026
* Deduplicate q4 quantization functions * Use const; add basic test * Re-enable quantization test * Disable AVX2 flags in CI --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
AlexiAlp
pushed a commit
to minghaop/llama.cpp
that referenced
this pull request
Jun 2, 2026
* Deduplicate q4 quantization functions * Use const; add basic test * Re-enable quantization test * Disable AVX2 flags in CI --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
ddvnguyen
pushed a commit
to ddvnguyen/llama.cpp
that referenced
this pull request
Jul 3, 2026
…gml-org#383 T2) T2 implements --peer-only: starts llama-engine without loading any model, just exposing the local GPU backend(s) as a ggml-RPC server and serving HTTP health checks. Replaces the nomic-embed-text workaround for the 3060 peer. Also fixes the start_shared_backend_rpc_server fallback path: when the scheduler has no non-CPU backends (e.g. tiny placeholder model), enumerate globally registered accelerator devices instead — same approach as rpc-server.cpp. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ddvnguyen
pushed a commit
to ddvnguyen/llama.cpp
that referenced
this pull request
Jul 3, 2026
3 tasks
hydra-z Bot
pushed a commit
to ddvnguyen/llama.cpp
that referenced
this pull request
Jul 5, 2026
…ra_vortex ggml-org#383) - --combined-split-mode layer for dense/SSM models: register peer RPC before model load, stock tensor_split places layers cleanly avoiding SSM-corruption - --peer-only engine mode: GPU backend + HTTP health, no model loaded - /slots returns empty array in peer-only mode (health check compatibility) - fit-params build target for VMM integration
This was referenced Jul 7, 2026
ddvnguyen
pushed a commit
to ddvnguyen/llama.cpp
that referenced
this pull request
Jul 7, 2026
The unified engine's model-loaded path hard-coded set_hydra_capabilities(... peer_reachable=false, combined_pattern="", split_mode="solo") even when it was launched with --rpc-engine PEER + --tensor-split, i.e. the peer was registered as an RPC device BEFORE load and llama.cpp's stock allocator placed whole layers across CUDA0 + RPC0 (COMBINED-static layer-split, ggml-org#383). Report the real state instead: - peer_reachable now reflects whether the peer actually registered (was always false, a reporting bug — /health and INFO claimed the peer was unreachable even while layers were split onto it). - split_mode="layer" when a peer registered before load with a tensor-split, so Hydra Core can route to the layer-split engine instead of the runtime expert path (the add-RPC-after-load path that causes the 35-graph-split regression + recurrent-state corruption on dense/SSM models). - combined_pattern kept EMPTY on purpose: it is the expert regex consumed by SET_EXPERT_MODE("combined") -> llama_hydra_rebind_combined_experts (server-context.cpp:3651); a non-empty value would trigger the after-load expert re-partition that layer-split exists to avoid. Observability/routing only; no compute-path change. Not compiled in this environment (no CUDA/2-GPU host) — needs a build_sm86_sm120 build to confirm. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012d7PtURptwKpgMcFZ2MvYn
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
As suggested in #356, this de-duplicates the code in
ggml_quantize_q4_0andggml_quantize_q4_1, which were recently moved to ggml.cTo ensure deterministic creation of model files, I introduced a new "reference" implementation for the q4_0 quantization. For q4_1 this wasn't necessary, as that has no SIMD optimizations.
This quashes @ggerganov's hope of making the quantize program faster, but I believe deterministic model files are more important.
Note that the checksum for
models/7B/ggml-model-q4_0.binis wrong inSHA256SUMS, see #374