Add EXAONE 4.5 implementations#45471
Conversation
There was a problem hiding this comment.
Hey @nuxlear , great addition!
I am seeing that the model is almost Qwen2-5VLVision + ExaoneLM with a small difference on the number of kv groups. Can you confirm if official ckpt have different values or if we can drop it, and fully copy from qwen?
If we can drop it, I left comment on how to clean it up further :)
Ah, and also, I see the doc page is missing which should be in docs/source/en/model_docs
| class Exaone4_5_TextConfig(Exaone4Config): | ||
| model_type = "exaone4_5_text" | ||
| base_config_key = "text_config" | ||
| keys_to_ignore_at_inference = ["past_key_values"] |
There was a problem hiding this comment.
this looks identical, we should be able to load Exaone4Config directly. For ex see how we load "llama" in llava by default
| from ..qwen2_vl.video_processing_qwen2_vl import Qwen2VLVideoProcessor | ||
|
|
||
|
|
||
| @strict |
There was a problem hiding this comment.
all configs have to be strict and also have @autodoctring(checkpoint="my-hub-repo")
zucchini-nlp
left a comment
There was a problem hiding this comment.
Nice work @nuxlear 🤩
I think the PR looks very good and we just need one final clean-up for style, before asing a core maintainer review
Ping me when you're ready and CI is green for model (ignore unrelated CI failures), and I will ask a core maintainer rveiew
| ("depth_anything", "DepthAnythingConfig"), | ||
| ("depth_pro", "DepthProConfig"), | ||
| ("detr", "DetrConfig"), | ||
| ("detr", "MaskFormerDetrConfig"), |
There was a problem hiding this comment.
was that done manually or after fix-repo 🤔 very weird if automatically, I need to check
There was a problem hiding this comment.
can you revert it, looks like a bad rebase
There was a problem hiding this comment.
It breaks after fix-repo in my env, so I fixed it manually.
There was a problem hiding this comment.
can you revert, CI isn't complaining in main branch, so i suppose it has smth to do with env setups. Can't point exactly what, so let's revert and see
actually, I have an idea why and I see duplicate names in mapping. Though it shouldn't complain and shouldn't block Exaone. In any case, fixing it needs to be a different PR
| outputs = self.model( | ||
| input_ids=input_ids, | ||
| pixel_values=pixel_values, | ||
| pixel_values_videos=pixel_values_videos, | ||
| image_grid_thw=image_grid_thw, | ||
| video_grid_thw=video_grid_thw, | ||
| second_per_grid_ts=second_per_grid_ts, | ||
| position_ids=position_ids, | ||
| attention_mask=attention_mask, | ||
| past_key_values=past_key_values, | ||
| inputs_embeds=inputs_embeds, | ||
| use_cache=use_cache, | ||
| **kwargs, | ||
| ) | ||
|
|
||
| hidden_states = outputs.last_hidden_state | ||
| slice_indices = slice(-logits_to_keep, None) if isinstance(logits_to_keep, int) else logits_to_keep | ||
| logits = self.lm_head(hidden_states[:, slice_indices, :]) | ||
|
|
||
| loss = None | ||
| if labels is not None: |
There was a problem hiding this comment.
we don't need forward to override a docsring, instead we can return super().forward(**super_kwargs)
There was a problem hiding this comment.
We don't use rope_deltas or mm_token_type_ids in forward(), so can we drop these kwargs and use CausalLMOutputWithPast instead of the generated Exaone4_5_CausalLMOutputWithPast? (which has rope_deltas unnecessarily)
There was a problem hiding this comment.
Ah I see, indeed, we don't have a way to easily drop args from signature
| class Exaone4_5_ProcessorKwargs(ProcessingKwargs, total=False): | ||
| _defaults = { | ||
| "text_kwargs": { | ||
| "padding": False, | ||
| }, | ||
| "videos_kwargs": {"return_metadata": True}, | ||
| } | ||
|
|
||
|
|
||
| class Exaone4_5_Processor(Qwen2_5_VLProcessor): | ||
| tokenizer_class = "AutoTokenizer" |
There was a problem hiding this comment.
nah, I see you added it in processing_auto, so now we can just delete these
There was a problem hiding this comment.
If I just delete Exaone4_5_Processor it breaks processing_exaone4_5.py
There was a problem hiding this comment.
we delete the file processing.py entirely, It is mapped in auto_processing to qwen,ii think
|
You might also need a rebase, and at last run |
|
@zucchini-nlp Sorry for the delay. I'm starting to address the feedback. BTW, since EXAONE 4.5 was recently released in vLLM (v0.20.0), we need to keep some class names as-is for compatibility, including Is there a recommended way to keep the model config unchanged while mapping the old name to the Transformers convention (e.g., If not, can I just patch this by aliasing the class name, e.g., |
|
@nuxlear not sure I understand the vLLM part. We didn't yet release the model in transformers, so vLLM should be using its own integration without importing anything like What part do we need to keep without changing? |
|
Yes, we understand that it would be better for vLLM to use its own integration. However, in this case, it would require updating our model config (e.g., So if there is a simple way to map or alias the existing config values and class names, we’d prefer that. Otherwise, we’d need to update the EXAONE 4.5 config, which would break compatibility with vLLM v0.20.0 and require additional changes. |
|
@nuxlear so vllm is importing smth that doesn't yet exist and isn't released? 🫠 In that case, I don't think it falls into breaking BC since it wasn't released yet and there is nothing to break. Seems like vLLM can't support Exaone before transformers release anyway |
|
@zucchini-nlp I understand. Then we will update the model config and open a patch PR later. I'll continue addressing the feedback 😃 |
|
@zucchini-nlp I think it's almost done, but I can't figure out why utils/check_repo.py is failing. It passes in my environment with the latest commit. (Edit: this was my bad. Never mind.) To get the tests fully passing, we need to update the config from |
zucchini-nlp
left a comment
There was a problem hiding this comment.
Great work, we can request a core maintainer review as last step before merging. I left some nitty-picky comments
Btw, are we deleting the unused processor class?
| processor = AutoProcessor.from_pretrained(model_id) | ||
| model = AutoModelForImageTextToText.from_pretrained( | ||
| model_id, | ||
| torch_dtype=torch.bfloat16, |
There was a problem hiding this comment.
nit: lets delete torch_dtype, we just merged a PR cleaning all docs. It default to dtype from config when loading
| ("depth_anything", "DepthAnythingConfig"), | ||
| ("depth_pro", "DepthProConfig"), | ||
| ("detr", "DetrConfig"), | ||
| ("detr", "MaskFormerDetrConfig"), |
There was a problem hiding this comment.
can you revert it, looks like a bad rebase
| self.num_key_value_groups = self.num_heads // self.num_key_value_heads | ||
| self.q_dim = self.num_heads * self.head_dim | ||
| self.kv_dim = self.num_key_value_heads * self.head_dim | ||
| self.qkv = nn.Linear(self.dim, self.q_dim + (self.kv_dim * 2), bias=True) |
There was a problem hiding this comment.
i was pointed recently, that we usually prefer unfused qkv, but I will leave it for core maintainer to decide. In any case, you won't have to change state-dicts, we will fuse-unfuse on the fly
There was a problem hiding this comment.
Honestly yes, but atp we have adopted too many qwen vl like checkpoints that have it fused. Usually it's easier on TP (unfused) but the vision backbones are small enough that it doesn't matter too much
| self, | ||
| hidden_states: torch.Tensor, | ||
| cu_seqlens: torch.Tensor, | ||
| rotary_pos_emb: torch.Tensor | None = None, |
There was a problem hiding this comment.
nit: unused arg rotary_pos_emb (I realize it is copy from qwen but lets delete)
| elif position_ids.ndim > 2: | ||
| position_ids = position_ids[-1] |
There was a problem hiding this comment.
is this possible? I think we shouldn't allow it and let it error out naturally at some point
| ) | ||
|
|
||
|
|
||
| @auto_docstring(checkpoint="LGAI-EXAONE/EXAONE-4.5-33B") |
There was a problem hiding this comment.
ultra nit: we don't need to add a ckpt everywhere, only in config classes :)
| class Exaone4_5_ProcessorKwargs(ProcessingKwargs, total=False): | ||
| _defaults = { | ||
| "text_kwargs": { | ||
| "padding": False, | ||
| }, | ||
| "videos_kwargs": {"return_metadata": True}, | ||
| } | ||
|
|
||
|
|
||
| class Exaone4_5_Processor(Qwen2_5_VLProcessor): | ||
| tokenizer_class = "AutoTokenizer" |
You mean the configs on the hub? Yes, looking at the tests looks like the changing model-type will fix it. If you cannot change # inlne comment here explaining why we override the model-type
if isinstance(text_config, dict):
model_type = text_config.get('model_type', 'exaone4')
if model_type = "exaone4_5_text": model_type = "exaone4"
text_config = CONFIG_MAPPING[model_type](**text_cofnig)
elif text_config is None:
text_config = CONFIG_MAPPING['exaone4']() |
|
failed test: https://app.circleci.com/pipelines/github/huggingface/transformers/173444/workflows/6dec560d-7daf-4af6-bac6-fb9973c822b6/jobs/2292655 |
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
CI ResultsCommit Info
Model CI Report❌ 2 new failed tests from this PR 😭
|
|
I will take a look in a bit 🤗 |
vasqu
left a comment
There was a problem hiding this comment.
Looks already very solid, left some comments but I think it's nothing major overall just smaller details
|
@vasqu Thanks for the review! Could you take a final look at the updates? |
|
[For maintainers] Suggested jobs to run (before merge) run-slow: auto, exaone4_5 |
|
run-slow: exaone4_5 |
vasqu
left a comment
There was a problem hiding this comment.
LGTM, I'm merging shortly just sanity checking with CI 🤗
|
This comment contains models: ["models/exaone4_5"] |
|
run-slow: exaone4_5 |
|
This comment contains models: ["models/exaone4_5"] |
|
Nice, gz @nuxlear glad we got this in 🫡 |
|
Finally! Thank you @vasqu and @zucchini-nlp for your help and effort 😄 |
* Add EXAONE 4.5 modeling & config code * Add test code for EXAONE 4.5 and Fix style * Address PR feedback * Address PR feedback * Update EXAONE 4.5 docs * Update docstring * Address the feedback * Fix docs * Minor fix * Minor fix * Update docs * Address the feedback * Skip failed test * Address the feedback * Minor fix * Revert skipping test * Fix test * Address the feedback * fixup autos, nits, and logits test to CI * fix date * last quick fixes --------- Co-authored-by: LG-AI-EXAONE <exaonemodels@lgresearch.ai> Co-authored-by: vasqu <antonprogamer@gmail.com> Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com>
# Add EXAONE 4.5 model support for Inference V2 ## Summary Add support for LG AI Research's EXAONE 4.5 language model in DeepSpeed Inference V2. The EXAONE 4.5 text decoder shares EXAONE 4.0's post-norm + QK-Norm parameter layout (`text_config.architectures == ["Exaone4ForCausalLM"]`), so the transformer container and most of the inference model are reused. EXAONE 4.5 is also a hybrid-attention model: `sliding_attention` layers apply llama3-scaled RoPE, and `full_attention` layers use no positional embedding (global NoPE). This implementation follows that reference behavior. ## Changes - New model implementation `deepspeed/inference/v2/model_implementations/exaone4_5/`: - `container.py`: non-transformer container for the multimodal checkpoint layout. The LM weights are nested under `model.language_model.` and `lm_head` stays top-level. - `model.py`: hybrid attention on top of the reused EXAONE 4.0 forward. - `sliding_attention` layers use trained-frequency RoPE with llama3-scaled inverse frequencies. A unit test checks the values against transformers' `ROPE_INIT_FUNCTIONS["llama3"]`. - `full_attention` layers dispatch to a separate NoPE attention module. - Sequence length is capped at `sliding_window` (4096). The dense blocked attention kernel has no local mask, and at or below the window size dense causal attention is equivalent to sliding attention. The cap is enforced in the scheduler path (`get_kv_requirements`) and the direct path (`maybe_allocate_kv`). - `activation_dtype` handles the transformers v5 config (`dtype` vs `torch_dtype`, str vs `torch.dtype`). - `policy.py`: extracts `text_config`, reuses `Exaone4TransformerContainer` under the `model.language_model.layers` prefix, and leaves the vision tower (`model.visual.`) and the MTP head (`mtp.`) unmapped. - `exaone4/model.py`: extracted a behavior-neutral `_forward_attention` seam for the per-layer dispatch, and fixed a latent aliasing crash in `_apply_qk_norm`. For a single-row slice, `contiguous()` returns an alias rather than a copy, so the write-back overlapped its own source and every single-token decode step raised a RuntimeError. The q/k slices are now cloned. This also affected EXAONE 4.0, which had not been exercised on the decode path. - `checkpoint/huggingface_engine.py`: derive `max_seq_length` from the nested `text_config.max_position_embeddings` when the top-level (multimodal) config doesn't have it. - `kernels/ragged_ops/linear_blocked_kv_rotary`: the trained-RoPE path now keeps the inverse frequencies and phase construction in FP32 instead of coupling them to the Q/K activation dtype (repair by @tohtana, merged from [Bias92#1](Bias92#1)). Q/K/V and the KV cache stay FP16/BF16; the rotated outputs are cast back to the activation dtype. A kernel test at position 4095 covers both dtypes. - Registered `exaone4_5` in `engine_factory.py` and `model_implementations/__init__.py`. - Unit tests for the llama3 frequency computation (against transformers), the per-layer attention dispatch, and the sequence-length cap. ## Why the mapping is exact Checked against the real `LGAI-EXAONE/EXAONE-4.5-33B` checkpoint index (1064 tensors). All 64 decoder layers expose exactly the 11 parameters the 4.0 container maps, and every non-language-model tensor falls under `model.visual.` (342) or `mtp.` (15), both declared unmapped. ## Testing Validated on an A100 80GB (SXM4) at `73c325e`, CUDA 12.4, torch 2.4.1+cu124 / transformers 5.13.1, with the real `LGAI-EXAONE/EXAONE-4.5-33B` checkpoint (bf16, shard SHA-256 verified): - Unit tests for the diff-impacted scope, run on CI-matched pytest 8.3.5: rotary kernels 58/58, blocked attention 23/23, EXAONE 4.5 model 6/6. - FP32-phase kernel test at position 4095 passes for both FP16 and BF16. - Load: both shards load and every container initializes (mapping and the `model.visual.`/`mtp.` skip both work). `max_sequence_length` resolves to the enforced 4096 cap and the runtime rope buffer is `torch.float32`. - All pre-commit hooks (including clang-format on the CUDA files) pass on the changed files. - Greedy generation through `engine.put` (single-sequence decode, which exercises the trained-rotary kernel, the NoPE module and the per-layer dispatch): ``` 'The capital of France is' -> ' Paris. \nThe capital of Germany is Berlin. \nThe capital of' '대한민국의 수도는' -> ' 어디인가요?\n서울입니다.\n질문은 대한민국의 수도를 묻고 있으며...' '2 + 2 =' -> ' 4.\n\nBut in the example, they said "the sum of all elements is' 'The opposite of hot is' -> ' cold. \nThe opposite of happy is sad. \nThe opposite of' ``` Note: the direct-put validation used `do_checks=False` because `engine.can_schedule` currently passes `state_manager.free_blocks` (a Python `list`, despite the `torch.Tensor` annotation) into `get_kv_requirements`, which expects an int. That is a pre-existing type mismatch that affects all models on the `put(do_checks=True)` path. MII drives the engine via `query()` + `put(do_checks=False)`, so it never hits it. Happy to file that separately. ## Scope Serves the base autoregressive text path of EXAONE 4.5 (e.g. `LGAI-EXAONE/EXAONE-4.5-33B`, bf16) up to 4096 tokens. Out of scope / follow-ups: - Vision tower (VLM, `Exaone4_5ForConditionalGeneration`) - MTP self-speculative decoding - FP8 / AWQ quantized variants (only fp16/bf16, same as EXAONE 4.0) - Long context (>4096), which needs a local attention mask in the dense blocked attention kernel The merged EXAONE 4.0 implementation has the same uniform-RoPE issue on its `full_attention` layers (plus unscaled llama3 rope). This PR keeps 4.0 behavior unchanged apart from the decode crash fix. I'd like to fix 4.0 in a follow-up PR so the two changes stay independently reviewable. ## Requirements `transformers >= 5.3.0`. EXAONE 4.5 landed in transformers 5.3 (huggingface/transformers#45471). --------- Signed-off-by: Bias92 <pewpewplay315@gmail.com> Signed-off-by: Masahiro Tanaka <mtanaka@anyscale.com> Co-authored-by: Masahiro Tanaka <mtanaka@anyscale.com>
What does this PR do?
Add EXAONE 4.5 architecture for the EXAONE 4.5 model released by LG AI Research.
This PR adds the modeling code for EXAONE 4.5, which uses the same LLM architecture as EXAONE 4.
Documentation will be updated.
Code Agent Policy
The Transformers repo is currently being overwhelmed by a large number of PRs and issue comments written by
code agents. We are currently bottlenecked by our ability to review and respond to them. As a result,
we ask that new users do not submit pure code agent PRs at this time.
You may use code agents in drafting or to help you diagnose issues. We'd also ask autonomous "OpenClaw"-like agents
not to open any PRs or issues for the moment.
PRs that appear to be fully agent-written will probably be closed without review, and we may block users who do this
repeatedly or maliciously.
This is a rapidly-evolving situation that's causing significant shockwaves in the open-source community. As a result,
this policy is likely to be updated regularly in the near future. For more information, please read
CONTRIBUTING.md.Before submitting
Pull Request section?
to it if that's the case.
documentation guidelines, and
here are tips on formatting docstrings.
Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.
@zucchini-nlp