mtmd : fix output-buffer size for multi-image batches without temporal merge#24634
mtmd : fix output-buffer size for multi-image batches without temporal merge#24634mayerwin wants to merge 1 commit into
Conversation
|
Hi @mayerwin, thanks for your contribution! Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:
Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below. |
…l merge mtmd_image_tokens::n_tokens() returned a single image's token count (nx*ny) whenever n_temporal_merge == 1, even when batch_f32 held multiple image entries. mtmd_encode_impl() uses n_tokens() to size the output embedding buffer, but clip_image_batch_encode() produces entries.size() * nx * ny tokens, so any request batching multiple images (e.g. two images in one message) aborted with "clip.cpp: Output buffer size mismatch". Reproduced with Gemma-4 vision. Multiply by the entry count when there is no temporal merge. The temporal-merge branch and the single-entry path are unchanged.
d0e6f07 to
0e7f29e
Compare
|
Disclosure, per the contribution guidelines: yes, I used an AI coding assistant on this, and I'd rather be upfront about that than hide it. What that meant in practice: the fix is four lines in The bug is real and reproducible on current |
|
the fix is incorrect, it will break qwen-vl model. I'll push another fix |
Summary
mtmd_image_tokens::n_tokens()under-counts the output tokens for a multi-image batchwhen the model does not use temporal merge, so
clip_image_batch_encode()aborts withOutput buffer size mismatch.Repro
Send a chat request with two images to a model whose image entries get batched into a
single
clip_image_batch_encodecall and that hasn_temporal_merge == 1(e.g. Gemma-4vision):
(
322560 = 1*126*2560,645120 = 2*126*2560— the buffer was sized for one image while theencoder produced two;
embeddings->ne[2] == entries.size() == 2.)Root cause
mtmd_encode_impl()sizes the output buffer asn_embd_out * image_tokens->n_tokens().n_tokens()returnsnx*ny(a single entry) whenevern_temporal_merge == 1, ignoringbatch_f32.entries.size(), butclip_image_batch_encode()producesentries.size() * nx * nytokens.Fix
Multiply by the entry count when there is no temporal merge. The single-entry path and the
temporal-merge branch (entries divided by the merge factor) are unchanged.
Testing
Built on Intel Arc (SYCL). Before: every 2-image Gemma-4 request aborts. After: 2-image
requests encode all entries and return normally; single-image paths (e.g. Qwen3-VL) unchanged.