[MultimodalLM] add language_model to the get/set_input_embeddings logic#46029
Conversation
| def get_input_embeddings(self): | ||
| return self.model.get_input_embeddings() | ||
|
|
||
| def set_input_embeddings(self, value): | ||
| self.model.set_input_embeddings(value) | ||
|
|
There was a problem hiding this comment.
language_model to EmbeddingAccessMixin. This was already redundant as get/set_input_embeddings already checks self.model
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
|
run-slow: aria, audioflamingo3, aya_vision, blip_2, cohere2_vision, deepseek_vl, deepseek_vl_hybrid, ernie4_5_vl_moe, exaone4_5, fast_vlm, florence2, fuyu, gemma3, gemma3n, gemma4, glm46v |
|
This comment contains models: ["models/aria", "models/audioflamingo3", "models/aya_vision", "models/blip_2", "models/cohere2_vision", "models/deepseek_vl", "models/deepseek_vl_hybrid", "models/ernie4_5_vl_moe", "models/exaone4_5", "models/fast_vlm", "models/florence2", "models/fuyu", "models/gemma3", "models/gemma3n", "models/gemma4", "models/glm46v"] |
vasqu
left a comment
There was a problem hiding this comment.
Nice pickup, small but huge QoL improvement 🤗 cc @molbap @zucchini-nlp just in case that you are aware
| model = getattr(self, "model", None) | ||
| if model is not None and hasattr(model, name): | ||
| return getattr(model, name) | ||
| # 4) Multimodal LMs that wrap a text sub-model at `self.language_model`. |
There was a problem hiding this comment.
Not on you but could you put a 5) below (similar to the set method)
There was a problem hiding this comment.
actually this is same as self.get_decoder() for all MLLMs, so we could
# Same as base model, we can check if embedding is inside a decoder for
# enc-dec models and for MLLMs
decoder = self.get_decoder()
embedding_layer = decoder.embedding_layer
CI ResultsCommit Info
Model CI Report❌ 1 new failed tests from this PR 😭
|
|
It's the process number 🫠 so all good it seems |
|
Perfect! merging |
|
[For maintainers] Suggested jobs to run (before merge) run-slow: aria, audioflamingo3, aya_vision, blip_2, cohere2_vision, deepseek_vl, deepseek_vl_hybrid, ernie4_5_vl_moe, exaone4_5, fast_vlm, florence2, fuyu, gemma3, gemma3n, gemma4, glm46v |
|
View the CircleCI Test Summary for this PR: https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=46029&sha=11f297 |
|
Flaky test merged myself 🤗 |
…ic (huggingface#46029) * add language_model to EmbeddingAccessMixin logic * propagate changes * fix test * add comment
…ic (huggingface#46029) * add language_model to EmbeddingAccessMixin logic * propagate changes * fix test * add comment
…ic (huggingface#46029) * add language_model to EmbeddingAccessMixin logic * propagate changes * fix test * add comment
What does this PR do?
As per-title, since we're checking self.model for encoder/decoder models, IMO there is no reason not to do it for ALM/VLMs too.