Skip to content

Modernize CLIP modeling code #41546

Merged
molbap merged 12 commits into
mainfrom
clip_improve
Oct 21, 2025
Merged

Modernize CLIP modeling code #41546
molbap merged 12 commits into
mainfrom
clip_improve

Conversation

@molbap

@molbap molbap commented Oct 13, 2025

Copy link
Copy Markdown
Collaborator

What does this PR do?

As per title. Adds the nice recent modeling utils to CLIP. Was motivated to be able to use it in the looong-standing #33962 .

@HuggingFaceDocBuilderDev

HuggingFaceDocBuilderDev commented Oct 13, 2025

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@yonigozlan yonigozlan mentioned this pull request Oct 13, 2025
Merged
@molbap

molbap commented Oct 20, 2025

Copy link
Copy Markdown
Collaborator Author

cc @zucchini-nlp if you want to take a look!

zucchini-nlp
zucchini-nlp approved these changes Oct 20, 2025
View reviewed changes

@zucchini-nlp zucchini-nlp left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Amazing, happy to see the models being standardized 🚀 Let's run slow tests with CLIP and LLaVA-CLIP before merging, just to be sure :)

@molbap

molbap commented Oct 20, 2025

Copy link
Copy Markdown
Collaborator Author

run-slow: clip, metaclip_2

@github-actions

github-actions Bot commented Oct 20, 2025

Copy link
Copy Markdown
Contributor

This comment contains run-slow, running the specified jobs:

models: ['models/clip', 'models/metaclip_2']
quantizations: [] ...

@molbap

molbap commented Oct 21, 2025

Copy link
Copy Markdown
Collaborator Author

run-slow: clip, metaclip_2

@github-actions

github-actions Bot commented Oct 21, 2025

Copy link
Copy Markdown
Contributor

[For maintainers] Suggested jobs to run (before merge)

run-slow: clip, metaclip_2

@github-actions

github-actions Bot commented Oct 21, 2025

Copy link
Copy Markdown
Contributor

This comment contains run-slow, running the specified jobs:

models: ['models/clip', 'models/metaclip_2']
quantizations: [] ...

ArthurZucker
ArthurZucker approved these changes Oct 21, 2025
View reviewed changes

@ArthurZucker ArthurZucker left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nine ty

@molbap
molbap merged commit 1d651c7 into main Oct 21, 2025
20 checks passed
@molbap
molbap deleted the clip_improve branch October 21, 2025 14:04
@vasqu vasqu mentioned this pull request Oct 21, 2025
Merged
ngazagna-qc pushed a commit to ngazagna-qc/transformers that referenced this pull request Oct 23, 2025
@molbap @ngazagna-qc
Modernize CLIP modeling code (huggingface#41546)
29bc4aa 
* stranded

* update modular

* modularities

* update

* fx broken

* fx stillb roken

* update

* missed this

* fix metaclip
i3hz pushed a commit to i3hz/transformers that referenced this pull request Oct 30, 2025
@molbap @i3hz
Modernize CLIP modeling code (huggingface#41546)
3f13209 
* stranded

* update modular

* modularities

* update

* fx broken

* fx stillb roken

* update

* missed this

* fix metaclip
SangbumChoi pushed a commit to SangbumChoi/transformers that referenced this pull request Jan 23, 2026
@molbap @SangbumChoi
Modernize CLIP modeling code (huggingface#41546)
8e0e7fe 
* stranded

* update modular

* modularities

* update

* fx broken

* fx stillb roken

* update

* missed this

* fix metaclip
@Tcc0403 Tcc0403 mentioned this pull request Feb 5, 2026
Merged
2 tasks
@yukiu00 yukiu00 mentioned this pull request Feb 5, 2026
Closed
4 tasks
@yukiu00

yukiu00 commented Feb 5, 2026

Copy link
Copy Markdown

Hi there,

We recently encountered an issue with transformers v5.0.0rc1 that seems to be related to the changes in this PR.

Specifically, CLIPVisionModel.forward() now returns hidden_states=None even when output_hidden_states=True is passed. It appears that the output_hidden_states argument was removed from ClipEncoder.forward (or is not being utilized from TransformersKwargs correctly).

Comparison:

We suspect this is a regression. Could you please check if this was intended?

Related

@ArthurZucker

ArthurZucker commented Feb 6, 2026

Copy link
Copy Markdown
Collaborator

Thanks for the report, having a look and yes it should work with check_model_inputs

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ArthurZucker ArthurZucker ArthurZucker approved these changes
@zucchini-nlp zucchini-nlp zucchini-nlp approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@molbap @HuggingFaceDocBuilderDev @yukiu00 @ArthurZucker @zucchini-nlp