quant: include MTP/NextN block when counting FFN layers (GLM-5.2)

[Deviad]

Fixes a quantization-time abort on models that carry a trailing Multi-Token-Prediction / NextN block (e.g. GLM-5.2), where llama-quantize fails with:

Bad layer 78 for tensor blk.78.ffn_down_shexp.weight. Must be in [0, 78)

Root cause

In init_quantize_state_counters() the per-FFN layer counters were seeded from hparams.n_layer():

qs.n_ffn_down = qs.n_ffn_gate = qs.n_ffn_up = n_layer();

But n_layer() is defined as:

n_layer() = n_layer_all - n_layer_nextn

i.e. it deliberately excludes the MTP/NextN block(s). For GLM-5.2 that means n_layer_all = 79, n_layer_nextn = 1, so n_layer() = 78 and the counters are set to 78.

The NextN block is still a real block in the file: it owns FFN tensors named blk.78.ffn_{down,gate,up}_shexp.weight. When the quantizer reaches those tensors, layer_info() parses the layer index out of the tensor name (78) and bounds-checks it against the counter with i_layer >= n_layer. Since 78 >= 78, it throws "Bad layer ...", aborting the whole quantization.

Fix

Seed the FFN counters from n_layer_all instead of n_layer() so the NextN block's layer index (== n_layer) is in range:

qs.n_ffn_down = qs.n_ffn_gate = qs.n_ffn_up = n_layer_all;

This only widens the valid [0, N) interval used for the name-based bounds check; it does not change which tensors are selected for the "more bits in the first/last eighth" heuristic in any way that affects the standard (non-NextN) layers, and it lets the NextN FFN tensors be quantized normally instead of aborting.

Refs: llama.cpp issue #24379

Overview

Additional information

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure:

[Deviad]

Here a couple of scripts I used.

#!/usr/bin/env bash
# Build a fresh llama.cpp from master with the glm-dsa arch + quantize tool.
# The homebrew llama.cpp (v9200) predates GLM-5.2 support and cannot load glm-dsa.
set -euo pipefail

SRC="${LLAMA_SRC:-$HOME/projects/llama.cpp}"
BUILD="$SRC/build-metal"
JOBS="${JOBS:-$(sysctl -n hw.ncpu)}"

echo "==> llama.cpp source: $SRC"
if [[ ! -d "$SRC/.git" ]]; then
  git clone --depth 1 https://github.com/ggml-org/llama.cpp "$SRC"
else
  echo "==> updating existing clone to master"
  git -C "$SRC" fetch --depth 1 origin master
  git -C "$SRC" checkout master
  git -C "$SRC" reset --hard origin/master
fi

# Verify the clone actually has GLM-5.2 support before building.
if ! grep -rq "GlmMoeDsa" "$SRC/conversion/glm.py" 2>/dev/null; then
  echo "FATAL: $SRC/conversion/glm.py has no GlmMoeDsa converter." >&2
  echo "       GLM-5.2 support is missing. Check your checkout." >&2
  exit 1
fi
echo "==> GlmMoeDsa converter present in source ✓"

echo "==> configuring cmake (Metal enabled, release) -> $BUILD"
cmake -S "$SRC" -B "$BUILD" \
  -DCMAKE_BUILD_TYPE=Release \
  -DGGML_METAL=ON \
  -DLLAMA_CURL=OFF \
  -DBUILD_SHARED_LIBS=OFF

echo "==> building llama-quantize + llama-gguf-split ($JOBS jobs)"
cmake --build "$BUILD" --config Release -j "$JOBS" \
  --target llama-quantize llama-gguf-split

LQ="$BUILD/bin/llama-quantize"
echo "==> built: $LQ"
"$LQ" --help 2>&1 | head -1

# Sanity: confirm this binary recognizes glm-dsa.
if strings "$LQ" 2>/dev/null | grep -qi "glm.moe.dsa\|glm-dsa"; then
  echo "==> glm-dsa arch recognized in binary ✓"
else
  echo "WARNING: glm-dsa string not found in binary — quantize may still work" >&2
fi

echo "==> DONE. llama-quantize ready at: $LQ"

#!/usr/bin/env bash
# Quantize GLM-5.2 to a custom mixed precision:
#   routed-expert MLP weights -> 2-bit (IQ2_S)
#   everything else           -> 4-bit (IQ4_NL base, preserved)
#
# Source: the existing Unsloth UD-IQ4_NL (9 shards, 347 GB).
# Output: new sharded GGUF (~9 shards, est ~240-260 GB).
set -euo pipefail

LLAMA_SRC="${LLAMA_SRC:-$HOME/projects/llama.cpp}"
LQ="$LLAMA_SRC/build-metal/bin/llama-quantize"

IN_DIR="/Volumes/Data NVME/GLM-5.2-GGUF/UD-IQ4_NL"
IN_SHARD="$IN_DIR/GLM-5.2-UD-IQ4_NL-00001-of-00009.gguf"

# Unsloth's importance matrix (used to create the UD-IQ4_NL source). Required
# for IQ2_* quant types. Downloaded from unsloth/GLM-5.2-GGUF repo root.
IMATRIX="/Volumes/Data NVME/GLM-5.2-GGUF/imatrix_unsloth.gguf"

OUT_DIR="/Volumes/Data NVME/GLM-5.2-GGUF/GLM-5.2-mixed-IQ2S-experts-IQ4NL-rest"
OUT_PREFIX="$OUT_DIR/GLM-5.2-mixed"
TENSOR_TYPES="/Volumes/Data NVME/GLM-5.2-GGUF/glm52_tensor_types.txt"

# --- 2-bit variant: override here if you want a different one ---
#   IQ2_S   (2.56 bpw)  <-- default, robust without imatrix
#   IQ2_M   (2.66 bpw)  best 2-bit quality
#   IQ2_XS  (2.31 bpw)  smaller; benefits from imatrix
#   IQ2_XXS (2.06 bpw)  smallest; benefits from imatrix
TWO_BIT="${TWO_BIT:-IQ2_S}"
BASE_BIT="${BASE_BIT:-IQ4_NL}"
NTHREADS="${NTHREADS:-28}"

if [[ ! -x "$LQ" ]]; then
  echo "FATAL: $LQ not found. Run ./build_llamacpp.sh first." >&2
  exit 1
fi
if [[ ! -f "$IN_SHARD" ]]; then
  echo "FATAL: input shard not found: $IN_SHARD" >&2
  exit 1
fi

# Regenerate the tensor-type file with the chosen 2-bit variant.
# IMPORTANT: llama.cpp applies tensor-type rules as regex_search() and FIRST
# MATCH WINS. Unsloth's imatrix has no entries for the MTP/NextN expert tensors
# in blk.78, so keep those at the high-precision base type and put those more
# specific rules before the generic routed-expert rules.
cat > "$TENSOR_TYPES" <<EOF
blk\\.78\\.ffn_down_exps=$BASE_BIT
blk\\.78\\.ffn_gate_exps=$BASE_BIT
blk\\.78\\.ffn_up_exps=$BASE_BIT
ffn_gate_exps=$TWO_BIT
ffn_up_exps=$TWO_BIT
ffn_down_exps=$TWO_BIT
EOF

mkdir -p "$OUT_DIR"

echo "==> source : $IN_DIR  (9 shards, IQ4_NL)"
echo "==> output : $OUT_DIR"
if [[ ! -f "$IMATRIX" ]]; then
  echo "FATAL: imatrix not found: $IMATRIX" >&2
  echo "  download: curl -L -o '$IMATRIX' https://huggingface.co/unsloth/GLM-5.2-GGUF/resolve/main/imatrix_unsloth.gguf_file" >&2
  exit 1
fi

echo "==> mapping: experts(3 fragments) -> $TWO_BIT | rest -> $BASE_BIT"
echo "==> imatrix: $IMATRIX"
echo "==> threads: $NTHREADS   keep-split: yes   allow-requantize: yes"
echo ""

time "$LQ" \
  --allow-requantize \
  --keep-split \
  --imatrix "$IMATRIX" \
  --tensor-type-file "$TENSOR_TYPES" \
  "$IN_SHARD" \
  "$OUT_PREFIX.gguf" \
  "$BASE_BIT" \
  "$NTHREADS"

echo ""
echo "==> DONE. Output shards:"
ls -lh "$OUT_DIR"/*.gguf
du -sh "$OUT_DIR"

[CISC]

Update OP AI disclosure please.

unwarranted changes

[CISC]

Please revert latest changes and address #24832 (review)