[generation] Encode multimodal data only once - #45783
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
|
run-slow: aria, aya_vision, blip_2, chameleon, cohere2_vision, deepseek_vl, deepseek_vl_hybrid, emu3, ernie4_5_vl_moe, fast_vlm, florence2, fuyu, gemma3, gemma3n, gemma4, glm46v |
|
run-slow: aria, aya_vision, blip_2, chameleon, cohere2_vision, deepseek_vl, deepseek_vl_hybrid, emu3, ernie4_5_vl_moe, fast_vlm, florence2, fuyu, gemma3, gemma3n, gemma4, glm46v, qwen2_vl, mllama, llava, llava_next_video |
|
This comment contains models: ["models/aria", "models/aya_vision", "models/blip_2", "models/chameleon", "models/cohere2_vision", "models/deepseek_vl", "models/deepseek_vl_hybrid", "models/emu3", "models/ernie4_5_vl_moe", "models/fast_vlm", "models/florence2", "models/fuyu", "models/gemma3", "models/gemma3n", "models/gemma4", "models/glm46v", "models/llava", "models/llava_next_video", "models/mllama", "models/qwen2_vl"] |
|
View the CircleCI Test Summary for this PR: https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=45783&sha=5c9599 |
CI recapDashboard: View test results in Grafana
|
|
[For maintainers] Suggested jobs to run (before merge) run-slow: aria, aya_vision, blip_2, chameleon, cohere2_vision, cosmos3_edge, cosmos3_omni, deepseek_vl, deepseek_vl_hybrid, emu3, ernie4_5_vl_moe, exaone4_5, fast_vlm, florence2, fuyu, gemma3 |
What does this PR do?
As per title, moves multimodal encoding out of the loop so it happens only once and on un-expanded pixels. For example, in assisted generation we used to encode twice (once in assistant and once in target model), or in beam search we used to expand all pixel by
x num_beamstimes. This is very inefficient, especially in models where vision encoder requires huge memory like qwen-series (usually extremely long sequence lengths)