[model] Support MiniCPM-V 4.0 by tc-mb · Pull Request #39899 · huggingface/transformers

tc-mb · 2025-08-05T03:16:32Z

This pull request supports the MiniCPM-V 4.0 model, which will be released soon.

…ed_list_of_images in ImageProcessor

…as the first sublist, not empty or None

zucchini-nlp · 2025-08-05T10:26:08Z

Great! LMK if you need an early review or would like the PR to be merged before the release

tc-mb · 2025-08-06T02:54:25Z

Great! LMK if you need an early review or would like the PR to be merged before the release

Well, we released this MiniCPM-V 4 a few hours ago. We want to merge it into transformers. We'd be happy to have your help.

zucchini-nlp · 2025-08-06T11:24:14Z

Yep, just saw, congrats on the release! Sure, I will do a preliminary review now and lmk if you need assistance with addressing comments/cleaning up the PR

zucchini-nlp

Hey, I feel like the code is pretty early-stage and doesn't fully align with transformers standards. I left comments below on what is expected to follow the mode addition standards

zucchini-nlp · 2025-08-06T11:58:42Z

        ("megatron-bert", "MegatronBertModel"),
        ("mgp-str", "MgpstrForSceneTextRecognition"),
        ("mimi", "MimiModel"),
+        ("minicpm_v_4", "MiniCPM_V_4Model"),


we need also to add ForConditionalGeneration in image-text-to-text mapping

zucchini-nlp · 2025-08-06T12:00:20Z

+class MiniCPMVSliceConfig(PretrainedConfig):
+    model_type = "minicpmv"
+
+    def __init__(
+        self,
+        patch_size=14,
+        max_slice_nums=9,
+        scale_resolution=448,
+        **kwargs,
+    ):
+        super().__init__(**kwargs)
+        self.patch_size = patch_size
+        self.max_slice_nums = max_slice_nums
+        self.scale_resolution = scale_resolution
+
+    @classmethod


Let's not add a new config class unless it can be used to init a model class. We can move it either under MiniCPMVisionConfig or MiniCPMConfig depending on where in model arch these attr are used

ok, now moved to MiniCPMVisionConfig

zucchini-nlp · 2025-08-06T12:02:07Z

+class MiniCPM_V_4Config(LlamaConfig):
+    model_type = "minicpmv"
+    keys_to_ignore_at_inference = ["past_key_values"]
+
+    default_vision_config = {
+        "hidden_size": 1152,
+        "image_size": 980,
+        "intermediate_size": 4304,


few comments here:

We can't use inheritance outside "modularXXX.py" files

The config has to be structured and contain a config.text_config, config.vision_config inside. So we need 3 config classes, where the text and vision configs can be just re-used from existing models. For ex, see LlavaConfig

Thanks for the suggestion. Now I regenerate configuration_xx.py and modeling_xx.py using the utils/modular_model_converter.py. This should hopefully be more aligned with the requirements.

zucchini-nlp · 2025-08-06T12:03:19Z

+import numpy as np
+import PIL
+import PIL.Image
+import PIL.ImageSequence
+import torch
+from PIL import Image


redundant imports

zucchini-nlp · 2025-08-06T12:03:30Z

+from transformers import AutoImageProcessor
+from transformers.image_processing_utils import BaseImageProcessor, BatchFeature
+from transformers.image_transforms import to_channel_dimension_format
+from transformers.image_utils import (
+    ChannelDimension,
+    ImageInput,
+    infer_channel_dimension_format,
+    is_batched,
+    is_torch_tensor,


let's use relative import here

zucchini-nlp · 2025-08-06T12:50:23Z

+from transformers.image_processing_utils import BatchFeature
+from transformers.image_utils import ImageInput
+from transformers.processing_utils import ProcessorMixin, ProcessingKwargs, Unpack, ImagesKwargs
+from transformers.tokenization_utils_base import (
+    PaddingStrategy,
+    PreTokenizedInput,
+    TextInput,


relative imports

ok, all imports now fixed

zucchini-nlp · 2025-08-06T12:51:47Z

+from transformers import LlamaTokenizerFast
+
+
+class MiniCPM_V_4TokenizerFast(LlamaTokenizerFast):
+    def __init__(self, **kwargs):
+        super().__init__(**kwargs)


we can just use LlamaTokenizer, no need to create a new tokenizer class. New special tokens can be added with https://huggingface.co/docs/transformers/en/main_classes/tokenizer#multimodal-tokenizer

zucchini-nlp · 2025-08-06T12:52:06Z

@@ -0,0 +1,45 @@
+from transformers import AutoModelForCausalLM, AutoModel, AutoTokenizer


to delete the file

zucchini-nlp · 2025-08-06T12:52:36Z

+
+if is_torch_available():
+    import torch
+


needs to have tests from common ModelTesterMixin and GenerationTesterMixin

zucchini-nlp · 2025-08-06T12:54:51Z

+    variance_scaling_(tensor, mode="fan_in", distribution="normal")
+
+
+class MiniCPMVisionEmbeddings(nn.Module):


can do MiniCPMVisionEmbeddings(SiglipVisionEmbeddings) and outline only the differences. Ig in this case the diff is in position ids

…ter.py

…placeholder to MiniCPM_V_4Processor

…odel

…rConditionalGeneration

…rainedModel -> MiniCPMVisionPreTrainedModel

…former(SiglipVisionTransformer)

…isionEmbeddings)

… logics outside to func get_image_features

…dule) && fix style

Regenerate configuration and modeling

zucchini-nlp · 2025-08-19T10:54:32Z

+            img_cnt = []
+            for pixel_values in pixel_values_list:
+                img_cnt.append(len(pixel_values))
+                all_pixel_values.extend([i.flatten(end_dim=1).permute(1, 0) for i in pixel_values])
+
+            # exist image
+            if all_pixel_values:
+                tgt_sizes = [tgt_size for tgt_size in tgt_sizes if isinstance(tgt_size, torch.Tensor)]
+                tgt_sizes = torch.vstack(tgt_sizes).type(torch.int32)
+
+                max_patches = torch.max(tgt_sizes[:, 0] * tgt_sizes[:, 1])
+
+                all_pixel_values = torch.nn.utils.rnn.pad_sequence(all_pixel_values, batch_first=True,
+                                                                   padding_value=0.0)
+                B, L, _ = all_pixel_values.shape
+                all_pixel_values = all_pixel_values.permute(0, 2, 1).reshape(B, 3, -1, L)
+
+                patch_attn_mask = torch.zeros((B, 1, max_patches), dtype=torch.bool, device=device)
+                for i in range(B):
+                    patch_attn_mask[i, 0, :tgt_sizes[i][0] * tgt_sizes[i][1]] = True
+


In continuation of discussion on MiniCPMBatchFeature:

I see we are padding pixels in any case when running the model, so i'd say we move do padding logic to image processors. That way we can use the default BatchFeature and it will be aligned with other UHD-like models

zucchini-nlp · 2025-08-19T10:55:27Z

+                if B > vision_batch_size:
+                    hs = []
+                    for i in range(0, B, vision_batch_size):
+                        start_idx = i
+                        end_idx = i + vision_batch_size
+                        tmp_hs = self.vpm(all_pixel_values[start_idx:end_idx], patch_attention_mask=patch_attn_mask[start_idx:end_idx], tgt_sizes=tgt_sizes[start_idx:end_idx]).last_hidden_state
+                        hs.append(tmp_hs)
+                    vision_embedding = torch.cat(hs, dim=0)
+                else:
+                    vision_embedding = self.vpm(all_pixel_values, patch_attention_mask=patch_attn_mask, tgt_sizes=tgt_sizes).last_hidden_state


We don't need to do batched run here, and instead just forward all pixels at once. Batching is handled either by Trainer or user's own inference code

zucchini-nlp · 2025-08-19T10:56:11Z

+            else: # no image
+                if self.training:
+                    dummy_image = torch.zeros(
+                        (1, 3, 224, 224),
+                        device=device, dtype=dtype
+                    )
+                    tgt_sizes = torch.Tensor([[(224 // self.config.patch_size), math.ceil(224 / self.config.patch_size)]]).type(torch.int32)
+                    dummy_feature = self.resampler(self.vpm(dummy_image).last_hidden_state, tgt_sizes)
+                else:
+                    dummy_feature = []
+                for _ in range(len(pixel_values_list)):
+                    vision_hidden_states.append(dummy_feature)


to delete, can't return random tensors if no images are present. The get_image_features methods can't be called without pixels

zucchini-nlp · 2025-08-19T10:57:30Z

+        if hasattr(self.llm.config, 'scale_emb'):
+            vllm_embedding = self.llm.model.embed_tokens(data['input_ids']) * self.llm.config.scale_emb
+        else:
+            vllm_embedding = self.llm.model.embed_tokens(data['input_ids'])


the config can always have scaling. If model doesn't need it, it will be set to 1.0 in config.json. Also we embed text ids in model's forward while self.get_image_features is reserved only to obtain processed image embeddings. Take a look at LLaVa model format

zucchini-nlp · 2025-08-19T10:59:36Z

+                    cur_vllm_emb.scatter_(0, image_indices.view(-1, 1).repeat(1, cur_vllm_emb.shape[-1]),
+                                          cur_vs_hs.view(-1, cur_vs_hs.shape[-1]))


Let's use out-of-place op here, and ig we could just get mask and masked_scatter once instead of iterating over each sample in batch

…fix some bugs in ModelTesterMixin

github-actions · 2025-09-01T03:58:21Z

[For maintainers] Suggested jobs to run (before merge)

run-slow: auto, minicpm_v_4

vasqu · 2026-06-15T13:26:06Z

What's the current state @tc-mb? 🤗

tc-mb · 2026-06-15T15:13:56Z

What's the current state @tc-mb? 🤗

This is too old; I'll get a new one when I have the chance.

cpp-qn added 11 commits July 15, 2025 14:54

init

f56b492

init

50bff84

fix format

7212ca2

fix format 1

440108e

update MiniCPM_V_4Processor call params

51620fb

update model code

a15c51d

update func _initialize_weights name in Resampler

db50ae7

update func batch_decode, decode in Processor, and usage of make_nest…

68b867e

…ed_list_of_images in ImageProcessor

after make_nested_list_of_images, all image sublist have same format …

9e93af6

…as the first sublist, not empty or None

add 2 test files

a79eeb8

delete unused imports in tests

7b45e9e

cpp-qn added 3 commits August 6, 2025 14:19

fix import in test

8db1509

try to fix most of ruff style format errors

b316d7d

fix ruff errors on from torch.nn.functional import *

e646a6c

zucchini-nlp reviewed Aug 6, 2025

View reviewed changes

cpp-qn added 12 commits August 8, 2025 11:46

Regenerate configuration and modeling with utils/modular_model_conver…

e7a9299

…ter.py

fix import bug in configuration_minicpm_v_4

ba0421d

move get_grid_placeholder, get_image_id_placeholder, get_slice_image_…

3042ad7

…placeholder to MiniCPM_V_4Processor

init MiniCPMVImageProcessorFast

a0aa40e

fix style

af204d4

add note on precision in fast image processor

7ad9dde

fix fast image processor style

fe27b49

fix comments on init_vision_module and init_resampler in MiniCPM_V_4M…

d116c96

…odel

delete code paths for training in get_vllm_embedding

4e19c84

fix MiniCPM_V_4Model forward func style

f5f0ed0

change old MiniCPM_V_4Model to new MiniCPM_V_4Model and MiniCPM_V_4Fo…

d2b3849

…rConditionalGeneration

fix 27 28 29 style

ad93a6f

cpp-qn and others added 10 commits August 14, 2025 18:02

remove _trunc_normal_, can be imported from torch

a2caac5

rename: MiniCPMVisionEncoder -> MiniCPMVisionEncoderLayer; SiglipPreT…

1ea0f07

…rainedModel -> MiniCPMVisionPreTrainedModel

MiniCPMVisionTransformer(SiglipPreTrainedModel) -> MiniCPMVisionTrans…

8614e37

…former(SiglipVisionTransformer)

delete test.py; fix imports in processing_minicpm_v_4

82c8ca4

MiniCPMVisionEmbeddings(nn.Module) -> MiniCPMVisionEmbeddings(SiglipV…

79ebc43

…isionEmbeddings)

update init_vision_module

30d244d

simplify get_vllm_embedding, move if vision_hidden_states not in data…

a5b18d7

… logics outside to func get_image_features

simplify Resampler params

51891e8

MultiheadAttention(nn.MultiheadAttention) -> MultiheadAttention(nn.Mo…

6d2d3f5

…dule) && fix style

Merge pull request #6 from tc-mb/try-support-minicpm_v_4

5cdfd71

Regenerate configuration and modeling

zucchini-nlp reviewed Aug 19, 2025

View reviewed changes

try_remove_MiniCPMVBatchFeature

e1b4096

tc-mb mentioned this pull request Aug 26, 2025

[model] Support MiniCPM-V 4.5 #40448

Open

init MiniCPM_V_4VisionText2TextModelTester and MiniCPM_V_4ModelTest; …

f4c41c8

…fix some bugs in ModelTesterMixin

evalstate mentioned this pull request Apr 29, 2026

Cumulative feature and defect updates from recent Transformers PRs evalstate/transformers#42

Open

tc-mb closed this Jun 15, 2026

		@@ -0,0 +1,45 @@
		from transformers import AutoModelForCausalLM, AutoModel, AutoTokenizer

		variance_scaling_(tensor, mode="fan_in", distribution="normal")


		class MiniCPMVisionEmbeddings(nn.Module):

		cur_vllm_emb.scatter_(0, image_indices.view(-1, 1).repeat(1, cur_vllm_emb.shape[-1]),
		cur_vs_hs.view(-1, cur_vs_hs.shape[-1]))

Uh oh!

Conversation

tc-mb commented Aug 5, 2025

Uh oh!

zucchini-nlp commented Aug 5, 2025

Uh oh!

tc-mb commented Aug 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zucchini-nlp commented Aug 6, 2025

Uh oh!

zucchini-nlp left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zucchini-nlp Aug 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Sep 1, 2025

Uh oh!

vasqu commented Jun 15, 2026

Uh oh!

tc-mb commented Jun 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

tc-mb commented Aug 6, 2025 •

edited

Loading

zucchini-nlp Aug 19, 2025 •

edited

Loading

tc-mb commented Jun 15, 2026 •

edited

Loading