Add MiniCPM5 tokenizer support#23384
Conversation
|
Hi @zhangtao2-1, thanks for your contribution! Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:
Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below. |
|
Since you're new here and MiniCPM 5 does not exist yet, cc/ @tc-mb We do not want to store the original regex in the GGUF, it should merely be accounted for in creating the Do not add the |
Good, but we still do not want to store the regex, see how other pre-tokenizers are handled in |
|
@CISC Removed storing pre-tokenizer regex in GGUF |
fb50971 to
8a1dc08
Compare
Add minicpm5 pre-tokenizer hash via convert_hf_to_gguf_update.py and implement hardcoded regex handling in llama-vocab.cpp, consistent with other BPE pre-tokenizers.
|
@CISC Removed the MiniCPM5-specific routing in llama.py per review — the default vocab fallback already reaches _set_vocab_gpt2(). |
|
@CISC Several CI jobs are red, but many fail at checkout with Your account is suspended (403). The PR-related checks (pre-tokenizer-hashes, lint, type-check) passed. Is there anything I should fix on my side, or can this wait for maintainer re-run? Thanks. |
Nope, everything is fine, it's just another GitHub outage. :) |
|
@CISC
|
PR is good to merge, just need to wait to GHA to be back up for Release. |
|
@CISC Is GHA restored yet, or still blocked? |
* origin/master: hexagon: add support for Q4_1 in MUL_MAT and MUL_MAT_ID (ggml-org#23647) ggml-webgpu: Fix how to dispatch WG to some ops (ggml-org#23750) vulkan: Switch MUL_MAT_VEC to 4 K per iteration for F16/32 (ggml-org#22887) vulkan: use GL_NV_cooperative_matrix_decode_vector for faster matmul (ggml-org#23541) vulkan: add REPEAT op support for f16 to f16. (ggml-org#23298) ci : move ARM jobs to self-hosted + disable kleidiai mac release (ggml-org#23780) vendor : update cpp-httplib to 0.46.0 (ggml-org#23650) pyproject : add conversion folder and update dependencies (ggml-org#23746) CUDA: restrict PDL to CTK >= 12.3 due to MSVC issues (ggml-org#23742) ci : bump cuda release to 13.3 (ggml-org#23749) common : fix env names to all have LLAMA_ARG_ prefix (ggml-org#23778) ci : fix windows ccaches (ggml-org#23777) ci : remove wasm test (ggml-org#23733) vulkan: avoid preferring transfer queue on AMD UMA devices (ggml-org#22455) ci : add ccache to server builds + fix undefined sanitizer build (ggml-org#23763) docs : fix duplicated "the" in granitevision and model-conversion docs (ggml-org#23767) convert: add MiniCPM5 tokenizer support (ggml-org#23384) server : fix the log message when using SSL (ggml-org#23393)
Add minicpm5 pre-tokenizer hash via convert_hf_to_gguf_update.py and implement hardcoded regex handling in llama-vocab.cpp, consistent with other BPE pre-tokenizers. Co-authored-by: zhangtao <zhangtao2@modelbest.cn>
Add minicpm5 pre-tokenizer hash via convert_hf_to_gguf_update.py and implement hardcoded regex handling in llama-vocab.cpp, consistent with other BPE pre-tokenizers. Co-authored-by: zhangtao <zhangtao2@modelbest.cn>
Add minicpm5 pre-tokenizer hash via convert_hf_to_gguf_update.py and implement hardcoded regex handling in llama-vocab.cpp, consistent with other BPE pre-tokenizers. Co-authored-by: zhangtao <zhangtao2@modelbest.cn>
Highlights:
- New canonical ROCmFPX runner: ghcr.io/hal0ai/hal0-rocmfpx:vulkan-minicpm5
(Hal0ai/Hal0_ROCmFPX@5b395660 + upstream PR #23384 cherry-pick for minicpm5
pre-tokenizer support). Single image serves ROCm/HIP and Vulkan backends.
Unblocks loading the BRAINTRAIN-1B GGUFs on Strix Halo.
- Slot-level image override: slot.image (top-level string in the slot TOML)
now overrides profile.image. Resolution order:
slot.image -> profile.image -> DEFAULT_ROCMFPX_IMAGE.
Future image bumps will be a code-only release (one constant in
schema.py); operators no longer have to chase profile clones.
Schema:
- Add DEFAULT_ROCMFPX_IMAGE = "ghcr.io/hal0ai/hal0-rocmfpx:vulkan-minicpm5"
constant in src/hal0/config/schema.py.
- The three ROCmFPX runner seed profiles (rocmfpx-rocm, vkfpx-moe,
vkfpx-dense) now reference vulkan-minicpm5.
- Add _resolve_image_ref(slot_cfg, profile) helper in
src/hal0/providers/container.py implementing the priority chain.
- _profile_image_and_flags now accepts slot_cfg and delegates image
resolution to _resolve_image_ref.
Tests:
- 11 new tests in tests/providers/test_image_resolution.py covering:
slot-level override (top-level + nested [slot] section), profile
fallback, DEFAULT_ROCMFPX_IMAGE fallback, image_gen dict collision
guard, empty-string override, and SEED_PROFILES image pinning.
Breaking:
- SEED_PROFILES for ROCmFPX runner changed from :server -> :vulkan-minicpm5.
- profile.image deprecated; will be removed from SEED_PROFILES in 0.9.6.
Refs:
- handoff: /tmp/handoff-b2-plan-2026-07-08.md (minicpm5 B2 research)
- handoff: /tmp/handoff-braintrain-nano-slot-2026-07-08.md (BRAINTRAIN context)
- upstream PR: ggml-org/llama.cpp#23384
Adds MiniCPM5 support for HF → GGUF conversion and inference.
Detect MiniCPM5 in LlamaModel and use the correct Llama3-style BPE + ByteLevel vocab path
Register the minicpm5 BPE pre-tokenizer fingerprint
Extract HF pre-tokenizer regexes into GGUF (tokenizer.ggml.pre_tokenizer_regex) and apply them at runtime via LLAMA_VOCAB_PRE_TYPE_CUSTOM_REGEX
Scoped to MiniCPM5 only; other Llama-family models are unchanged.