🚨 🚨Bring some dinos to modern standards by molbap · Pull Request #46266 · huggingface/transformers

molbap · 2026-05-28T16:22:48Z

What does this PR do?

Part of the larger vision model refactor #41693 focused on dinov2, which has still some usage and downloads, but mostly serves as a basis for many other models. Attempt at putting this in line with the rest of the lib.

HuggingFaceDocBuilderDev · 2026-05-28T16:43:04Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

guarin

Thanks, this will make things much easier! Left more questions than comments :)

guarin

Nice refactor! Looks mostly good to me, left some comments and questions :)

vasqu

Only took a look at dinov2, I think we need to make sure this is good and then we can propogate based on that.

Sorry for the confusion on some comments, please read everything before responding 🫠

vasqu · 2026-06-10T18:45:14Z

+    pass
+
+
+class Dinov2Embeddings(nn.Module):


Any reason we don't inherit from e.g. BeitEmbeddings. Not convinced that we cannot use modular here

I couldv'e sworn I had answered these 👁️
however it's a bit tricky, it can change logits a bit due to upcasting. Trying a run-slow though

vasqu · 2026-06-10T18:48:14Z

+        return hidden_state * self.lambda1
+
+
+class Dinov2MLP(nn.Module):


Imo can be CLIPMLP for the forward

vasqu · 2026-06-10T18:50:36Z

+
+    def __init__(self, config: Dinov2Config) -> None:
+        super().__init__()
+        self.norm1 = nn.LayerNorm(config.hidden_size, eps=config.layer_norm_eps)


Imo, we could copy llama here and add the layer_scales (which are the only new thing)

Inheriting LlamaDecoderLayer means overriding both init and forward, beyond the layer scales we'd also be swapping RMSNorm→LayerNorm and dropping all the cache/RoPE 😬

Makes sense, always annoying 😢

vasqu · 2026-06-10T18:53:45Z

+        self.embeddings = Dinov2Embeddings(config)
+        self.encoder = Dinov2Encoder(config)
+        self.layernorm = nn.LayerNorm(config.hidden_size, eps=config.layer_norm_eps)
+        self.pooler = None


Suggested change

self.pooler = None

vasqu · 2026-06-10T18:57:33Z

+    Dinov2 backbone, to be used with frameworks like DETR and MaskFormer.
+    """
+)
+class Dinov2Backbone(BackboneMixin, Dinov2PreTrainedModel):


No similar backbone to use?

unfortunately, not really! (I think answered elsewhere but worth adding back)

…nto improve_dinos

molbap · 2026-06-12T15:17:06Z

run-slow: depth_anything, dinov2, dinov2_with_registers, dinov3_convnext, dinov3_vit, eomt, eomt_dinov3, pixio, rf_detr, sapiens2, videomt

github-actions · 2026-06-12T15:18:28Z

Workflow Run ⚙️

This comment contains run-slow, running the specified jobs:

models: ["models/depth_anything", "models/dinov2", "models/dinov2_with_registers", "models/dinov3_convnext", "models/dinov3_vit", "models/eomt", "models/eomt_dinov3", "models/pixio", "models/rf_detr", "models/sapiens2", "models/videomt"]
quantizations: []

github-actions · 2026-06-12T15:39:56Z

CI Results

Workflow Run ⚙️

Commit Info

Context	Commit	Description
RUN	42daf802	workflow commit (merge commit)
PR	4ee1242a	branch commit (from PR)
main	f208766a	base commit (on `main`)

✅ No failing test specific to this PR 🎉 👏 !

github-actions · 2026-06-15T16:29:05Z

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=46266&sha=b0ea71

…nto improve_dinos

molbap · 2026-06-26T16:10:06Z

@guarin if you can have another look, I updated the swiglu-related reverse load test, I think it's an acceptable workaround. LMK!

guarin

Nice refactor! Just some minor points/questions here and there, otherwise LGTM!

guarin · 2026-06-29T13:25:02Z

+        patch_size = self.patch_size if isinstance(self.patch_size, Iterable) else (self.patch_size, self.patch_size)
+        new_height = height // patch_size[0]
+        new_width = width // patch_size[1]



Would it be possible to already create self.patch_size as a tuple? Or does this break BC?

yeah I think it breaks BC here, in particular for rf_detr IIRC

guarin · 2026-06-29T13:29:35Z

+        image_size = config.image_size
+        patch_size = config.patch_size
+        image_size = image_size if isinstance(image_size, Iterable) else (image_size, image_size)
+        patch_size = patch_size if isinstance(patch_size, Iterable) else (patch_size, patch_size)


Nit: Inline to self.image_size = config.image_size if isinstance(config.image_size, Iterable) else (config.image_size, config.image_size) to allow overwriting self.image_size and self.patch_size with modular?

This is coming from VitPatchEmbeddings IIRC, so it's not modifiable directly

guarin · 2026-06-29T13:33:57Z

+        down_proj = self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
+        return down_proj


Suggested change

down_proj = self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))

return down_proj

return self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))

eh, that's llama-inherited 😬

guarin · 2026-06-29T13:35:42Z

    _supports_attention_backend = True
+    _can_compile_fullgraph = True


guarin · 2026-06-29T13:39:07Z

        ```"""
-        kwargs["output_hidden_states"] = True  # required to extract layers for the stages
-
+        kwargs["output_hidden_states"] = True  # required to extract per-stage feature maps from hidden_states


Could this be replaced by a decorator?

AH indeed, I think #46422 (that I merged...) covers it. checking

guarin · 2026-06-29T13:42:29Z

+                    patch_size = (
+                        self.config.patch_size
+                        if isinstance(self.config.patch_size, Iterable)
+                        else (self.config.patch_size, self.config.patch_size)
+                    )


Looking at all these I wonder whether we don't want to convert to tuple in config __post_init__. Not sure if that works though if users set config.patch_size = 14 though.

Not sure if that works though if users set config.patch_size = 14 though.

Yes, I think it'll break unfortunately

guarin · 2026-06-29T13:44:59Z


 class Dinov2WithRegistersEmbeddings(nn.Module):
-    """
-    Construct the CLS token, mask token, register tokens, position and patch embeddings.


Could we inherit from Dinov2Embeddings?

we can, but it's a lot of overrides, so the reuse isn't bringing much IMO

Imo we can inherit for the init? And highlight the diffs with small comments

guarin · 2026-06-30T10:58:01Z


-        # Validate output dimensions if not tracing
        if not torch.jit.is_tracing():
            if int(height) != patch_pos_embed.shape[-2] or int(width) != patch_pos_embed.shape[-1]:


Is there a particular reason why we have this check? Because it essentially checks if nn.functional.interpolate resized the embeddings to the correct size which seems redundant.

uh indeed that seems to be a dead check. The point of these is to guarantee static shapes/force interpolate during tracing, which is what it does on other models, but is not useful/wrongly placed here

github-actions · 2026-06-30T14:08:30Z

[For maintainers] Suggested jobs to run (before merge)

run-slow: depth_anything, dinov2, dinov2_with_registers, dinov3_convnext, dinov3_vit, eomt, eomt_dinov3, pixio, rf_detr, sapiens2, videomt

molbap · 2026-06-30T14:14:45Z

run-slow: depth_anything, dinov2, dinov2_with_registers, dinov3_convnext, dinov3_vit, eomt, eomt_dinov3, pixio, rf_detr, sapiens2, videomt

github-actions · 2026-06-30T14:16:38Z

Workflow Run ⚙️

This comment contains run-slow, running the specified jobs:

models: ["models/depth_anything", "models/dinov2", "models/dinov2_with_registers", "models/dinov3_convnext", "models/dinov3_vit", "models/eomt", "models/eomt_dinov3", "models/pixio", "models/rf_detr", "models/sapiens2", "models/videomt"]
quantizations: []

github-actions · 2026-06-30T14:21:38Z

CI recap

Dashboard: View test results in Grafana
Latest run: 28450557810:1
Result: failure | Jobs: 14 | Tests: 50,911 | Failures: 0 | Duration: 14h 37m

github-actions · 2026-06-30T18:06:52Z

CI Results

Workflow Run ⚙️

Commit Info

Context	Commit	Description
RUN	d28e35e5	workflow commit (merge commit)
PR	a3aa0d0e	branch commit (from PR)
main	181beb3b	base commit (on `main`)

✅ No failing test specific to this PR 🎉 👏 !

vasqu · 2026-07-01T10:43:58Z

Let me know if I should make a last pass 🤗

molbap · 2026-07-01T12:47:50Z

No rush, but glad to have a review if you have time! I iterated mostly on dinov2 but the rest seems OK as well @vasqu

vasqu

Looking super solid, I have a few smaller comments but nothing major. I think we are close to getting this merged 🫡

vasqu · 2026-07-17T15:09:34Z

+    def __init__(self, config: Dinov2Config) -> None:
+        nn.Module.__init__(self)
+        self.cls_token = nn.Parameter(torch.randn(1, 1, config.hidden_size))
+        self.mask_token = nn.Parameter(torch.zeros(1, config.hidden_size)) if config.use_mask_token else None
+        self.patch_embeddings = Dinov2PatchEmbeddings(config)
+        self.position_embeddings = nn.Parameter(
+            torch.randn(1, self.patch_embeddings.num_patches + 1, config.hidden_size)
+        )
+        self.dropout = nn.Dropout(config.hidden_dropout_prob)
+        self.patch_size = config.patch_size
+        self.config = config


Suggested change

def __init__(self, config: Dinov2Config) -> None:

nn.Module.__init__(self)

self.cls_token = nn.Parameter(torch.randn(1, 1, config.hidden_size))

self.mask_token = nn.Parameter(torch.zeros(1, config.hidden_size)) if config.use_mask_token else None

self.patch_embeddings = Dinov2PatchEmbeddings(config)

self.position_embeddings = nn.Parameter(

torch.randn(1, self.patch_embeddings.num_patches + 1, config.hidden_size)

)

self.dropout = nn.Dropout(config.hidden_dropout_prob)

self.patch_size = config.patch_size

self.config = config

def __init__(self, config: Dinov2Config) -> None:

super().__init__(self, config)

del num_patches

self.position_embeddings = nn.Parameter(

torch.randn(1, self.patch_embeddings.num_patches + 1, config.hidden_size)

)

can we not simplify a bit?

vasqu · 2026-07-17T15:11:54Z

+        patch_size = self.patch_size if isinstance(self.patch_size, Iterable) else (self.patch_size, self.patch_size)
+        new_height = height // patch_size[0]
+        new_width = width // patch_size[1]
+
+        sqrt_num_positions = torch_int(num_positions**0.5)
+        patch_pos_embed = patch_pos_embed.reshape(1, sqrt_num_positions, sqrt_num_positions, dim)
+        patch_pos_embed = patch_pos_embed.permute(0, 3, 1, 2)
+        target_dtype = patch_pos_embed.dtype
+        patch_pos_embed = nn.functional.interpolate(
+            patch_pos_embed.to(torch.float32),
+            size=(new_height, new_width),
+            mode="bicubic",
+            align_corners=False,
+        ).to(dtype=target_dtype)


Ok so I see 2 key differences

The patch size can be different for width and height

Dtype casting around interpolation

Let's highlight those with small comments

vasqu · 2026-07-17T15:14:34Z

+        nn.Module.__init__(self)
+        in_features = out_features = config.hidden_size
+        hidden_features = int(config.hidden_size * config.mlp_ratio)
+        self.fc1 = nn.Linear(in_features, hidden_features, bias=True)
+        self.activation_fn = ACT2FN[config.hidden_act]
+        self.fc2 = nn.Linear(hidden_features, out_features, bias=True)


Suggested change

nn.Module.__init__(self)

in_features = out_features = config.hidden_size

hidden_features = int(config.hidden_size * config.mlp_ratio)

self.fc1 = nn.Linear(in_features, hidden_features, bias=True)

self.activation_fn = ACT2FN[config.hidden_act]

self.fc2 = nn.Linear(hidden_features, out_features, bias=True)

super().__init__(self, config)

# Key difference: intermediate size is calculated based on ratio

self.fc1 = nn.Linear(config.hidden_size, int(config.hidden_size * config.mlp_ratio))

self.fc2 = nn.Linear(int(config.hidden_size * config.mlp_ratio), config.hidden_size)

Imo we can even fully copy based on adding a post init and intermediate size attr in the config

vasqu · 2026-07-17T15:16:12Z

+        nn.Module.__init__(self)
+        hidden_features = int(config.hidden_size * config.mlp_ratio)
+        hidden_features = (int(hidden_features * 2 / 3) + 7) // 8 * 8
+        self.gate_proj = nn.Linear(config.hidden_size, hidden_features, bias=True)
+        self.up_proj = nn.Linear(config.hidden_size, hidden_features, bias=True)
+        self.down_proj = nn.Linear(hidden_features, config.hidden_size, bias=True)
+        self.act_fn = nn.functional.silu


Same here, ok but this will break with different intermediate sizes? maybe can be a conditional that calculates it based on whether swiglu is used or not

Is the activation only used in the mlp? Imo, we can also override the activation in post init when we use swiglu

vasqu · 2026-07-17T15:17:08Z

+
+    def __init__(self, config: Dinov2Config) -> None:
+        super().__init__()
+        self.norm1 = nn.LayerNorm(config.hidden_size, eps=config.layer_norm_eps)


Makes sense, always annoying 😢

vasqu · 2026-07-17T15:25:20Z

+        hidden_state = embedding_output
+        hidden_states = (hidden_state,)
+        for layer in self.encoder.layer:
+            hidden_state = layer(hidden_state, **kwargs)
+            hidden_states = hidden_states + (hidden_state,)


hmm what happened here that we don't have an encoder here? That was more elegant no?

vasqu · 2026-07-17T15:25:33Z

-            hidden_states=hidden_states,
-            attentions=output.attentions,
-        )
+        return BackboneOutput(feature_maps=tuple(feature_maps))


Still tuple cast needed?

vasqu · 2026-07-17T15:25:47Z

        super()._init_weights(module)
        if hasattr(module, "segmentation_bias") and isinstance(module.segmentation_bias, nn.Parameter):
-            nn.init.constant_(module.segmentation_bias, 0.0)
+            init.constant_(module.segmentation_bias, 0.0)


Oh wow how did we miss this yikes

vasqu · 2026-07-17T15:27:09Z

-        expected_slice = torch.tensor(
-            [[8.8223, 8.6483, 8.6216], [8.3332, 8.6047, 8.7545], [8.6547, 8.6885, 8.7472]],
-        ).to(torch_device)
+        # Re-baselined after the dinov2 refactor adopted ViT's fp32-softmax `eager_attention_forward`.


Hmm so it didnt have fp32 before in the softmax? Maybe we should revert on that and keep it as close as we can to the original. Pretty sure there is a eager attn with no fp32 softmax like bart

vasqu · 2026-07-17T15:27:26Z

+        # Re-baselined after the dinov2 refactor adopted ViT's fp32-softmax `eager_attention_forward`.
+        slice_expectations = Expectations(
+            {
+                ("cpu", None): [[8.8223, 8.6483, 8.6215], [8.3332, 8.6047, 8.7545], [8.6547, 8.6885, 8.7472]],


Since when do we run on cpu 👀

molbap added 5 commits May 28, 2026 14:40

First draft + todo

94f8c0e

propagate changes, fix tests

c3c3b3a

mask token?

1978ae7

conversion bug

feea8f6

update decorators again, keep Encoders

8743081

guarin reviewed May 29, 2026

View reviewed changes

molbap mentioned this pull request May 29, 2026

Add Sapiens2 Model #45919

Merged

39 tasks

molbap added 5 commits June 1, 2026 17:24

simplifications

8acf60f

attention mask

2b3b370

registers conversion

e1191d9

Merge branch 'main' into improve_dinos

6a0a80d

fixup merge

b512098

molbap marked this pull request as ready for review June 4, 2026 13:04

molbap changed the title ~~Improve dinos~~ 🚨 🚨Bring some dinos to modern standards Jun 4, 2026

molbap added 3 commits June 4, 2026 18:14

fall back to usual hidden states routing for now

1e29c49

init

0173beb

Merge branch 'main' into improve_dinos

76df3ae

guarin reviewed Jun 9, 2026

View reviewed changes

Comment thread src/transformers/models/dinov2/modeling_dinov2.py

guarin reviewed Jun 9, 2026

View reviewed changes

molbap added 6 commits June 9, 2026 12:09

remove redundant dec

561928e

other fixes

486465e

remove manual setting

50ce0e4

re-re-revert logic on backbone output tracing 🫪🫪🫪

351cc6c

rf_detr changes

cb9b995

woops, wromg PR

555993a

vasqu reviewed Jun 10, 2026

View reviewed changes

molbap and others added 3 commits June 12, 2026 10:04

dinov2 simplifications

bd67b16

modular

bbda225

Merge branch 'main' into improve_dinos

30bd179

molbap added 2 commits June 12, 2026 16:53

modularize more?

a78b308

Merge branch 'improve_dinos' of github.com:huggingface/transformers i…

4ee1242

…nto improve_dinos

molbap and others added 3 commits June 12, 2026 18:38

boom

0872924

better patch size and conversion

bb8390d

Merge branch 'main' into improve_dinos

b0ea713

molbap added 3 commits June 26, 2026 18:00

fixup tests

fdfab53

Merge remote-tracking branch 'origin/main' into improve_dinos

c08d497

Merge branch 'improve_dinos' of github.com:huggingface/transformers i…

3e92e95

…nto improve_dinos

molbap requested a review from guarin June 26, 2026 16:08

guarin approved these changes Jun 29, 2026

View reviewed changes

remove hidden states tracking

d72df9f

guarin reviewed Jun 30, 2026

View reviewed changes

dead check

a3aa0d0

vasqu reviewed Jul 17, 2026

View reviewed changes

		return hidden_state * self.lambda1


		class Dinov2MLP(nn.Module):

		down_proj = self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
		return down_proj

	down_proj = self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
	return down_proj
	return self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))

		_supports_attention_backend = True
		_can_compile_fullgraph = True

Uh oh!

Conversation

molbap commented May 28, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Uh oh!

HuggingFaceDocBuilderDev commented May 28, 2026

Uh oh!

guarin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

guarin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vasqu left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

molbap commented Jun 12, 2026

Uh oh!

github-actions Bot commented Jun 12, 2026

Uh oh!

github-actions Bot commented Jun 12, 2026

CI Results

Commit Info

Uh oh!

github-actions Bot commented Jun 15, 2026

Uh oh!

molbap commented Jun 26, 2026

Uh oh!

guarin left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

molbap commented May 28, 2026 •

edited by github-actions Bot

Loading

molbap Jun 30, 2026 •

edited

Loading