Skip to content

Add EXAONE 4.5 implementations#45471

Merged
vasqu merged 22 commits into
huggingface:mainfrom
nuxlear:add-exaone4_5
May 4, 2026
Merged

Add EXAONE 4.5 implementations#45471
vasqu merged 22 commits into
huggingface:mainfrom
nuxlear:add-exaone4_5

Conversation

@nuxlear

@nuxlear nuxlear commented Apr 16, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

Add EXAONE 4.5 architecture for the EXAONE 4.5 model released by LG AI Research.

This PR adds the modeling code for EXAONE 4.5, which uses the same LLM architecture as EXAONE 4.
Documentation will be updated.

Code Agent Policy

The Transformers repo is currently being overwhelmed by a large number of PRs and issue comments written by
code agents. We are currently bottlenecked by our ability to review and respond to them. As a result,
we ask that new users do not submit pure code agent PRs at this time.
You may use code agents in drafting or to help you diagnose issues. We'd also ask autonomous "OpenClaw"-like agents
not to open any PRs or issues for the moment.

PRs that appear to be fully agent-written will probably be closed without review, and we may block users who do this
repeatedly or maliciously.

This is a rapidly-evolving situation that's causing significant shockwaves in the open-source community. As a result,
this policy is likely to be updated regularly in the near future. For more information, please read CONTRIBUTING.md.

  • I confirm that this is not a pure code agent PR.

Before submitting

  • This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
  • Did you read the contributor guideline,
    Pull Request section?
  • Was this discussed/approved via a Github issue or the forum? Please add a link
    to it if that's the case.
  • Did you make sure to update the documentation with your changes? Here are the
    documentation guidelines, and
    here are tips on formatting docstrings.
  • Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@zucchini-nlp

@zucchini-nlp zucchini-nlp left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @nuxlear , great addition!

I am seeing that the model is almost Qwen2-5VLVision + ExaoneLM with a small difference on the number of kv groups. Can you confirm if official ckpt have different values or if we can drop it, and fully copy from qwen?

If we can drop it, I left comment on how to clean it up further :)

Ah, and also, I see the doc page is missing which should be in docs/source/en/model_docs

Comment on lines +48 to +51
class Exaone4_5_TextConfig(Exaone4Config):
model_type = "exaone4_5_text"
base_config_key = "text_config"
keys_to_ignore_at_inference = ["past_key_values"]

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this looks identical, we should be able to load Exaone4Config directly. For ex see how we load "llama" in llava by default

https://github.com/huggingface/transformers/blob/main/src/transformers/models/llava/configuration_llava.py

from ..qwen2_vl.video_processing_qwen2_vl import Qwen2VLVideoProcessor


@strict

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

all configs have to be strict and also have @autodoctring(checkpoint="my-hub-repo")

Comment thread src/transformers/models/exaone4_5/modular_exaone4_5.py Outdated
Comment thread src/transformers/models/exaone4_5/modular_exaone4_5.py Outdated
Comment thread src/transformers/models/exaone4_5/modular_exaone4_5.py Outdated
Comment thread src/transformers/models/exaone4_5/modular_exaone4_5.py Outdated
Comment thread src/transformers/models/exaone4_5/modular_exaone4_5.py
Comment thread src/transformers/models/exaone4_5/modular_exaone4_5.py Outdated
Comment thread src/transformers/models/exaone4_5/modular_exaone4_5.py Outdated
Comment thread tests/models/exaone4_5/test_modeling_exaone4_5.py

@zucchini-nlp zucchini-nlp left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work @nuxlear 🤩

I think the PR looks very good and we just need one final clean-up for style, before asing a core maintainer review

Ping me when you're ready and CI is green for model (ignore unrelated CI failures), and I will ask a core maintainer rveiew

Comment thread docs/source/en/model_doc/exaone4_5.md Outdated
Comment thread docs/source/en/model_doc/exaone4_5.md Outdated
Comment thread docs/source/en/model_doc/exaone4_5.md
Comment thread docs/source/ko/model_doc/exaone4_5.md
("depth_anything", "DepthAnythingConfig"),
("depth_pro", "DepthProConfig"),
("detr", "DetrConfig"),
("detr", "MaskFormerDetrConfig"),

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

was that done manually or after fix-repo 🤔 very weird if automatically, I need to check

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you revert it, looks like a bad rebase

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It breaks after fix-repo in my env, so I fixed it manually.

@zucchini-nlp zucchini-nlp Apr 29, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you revert, CI isn't complaining in main branch, so i suppose it has smth to do with env setups. Can't point exactly what, so let's revert and see

actually, I have an idea why and I see duplicate names in mapping. Though it shouldn't complain and shouldn't block Exaone. In any case, fixing it needs to be a different PR

Comment thread src/transformers/models/exaone4_5/modular_exaone4_5.py Outdated
Comment thread src/transformers/models/exaone4_5/modular_exaone4_5.py Outdated
Comment on lines +449 to +469
outputs = self.model(
input_ids=input_ids,
pixel_values=pixel_values,
pixel_values_videos=pixel_values_videos,
image_grid_thw=image_grid_thw,
video_grid_thw=video_grid_thw,
second_per_grid_ts=second_per_grid_ts,
position_ids=position_ids,
attention_mask=attention_mask,
past_key_values=past_key_values,
inputs_embeds=inputs_embeds,
use_cache=use_cache,
**kwargs,
)

hidden_states = outputs.last_hidden_state
slice_indices = slice(-logits_to_keep, None) if isinstance(logits_to_keep, int) else logits_to_keep
logits = self.lm_head(hidden_states[:, slice_indices, :])

loss = None
if labels is not None:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we don't need forward to override a docsring, instead we can return super().forward(**super_kwargs)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't use rope_deltas or mm_token_type_ids in forward(), so can we drop these kwargs and use CausalLMOutputWithPast instead of the generated Exaone4_5_CausalLMOutputWithPast? (which has rope_deltas unnecessarily)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah I see, indeed, we don't have a way to easily drop args from signature

Comment on lines +520 to +530
class Exaone4_5_ProcessorKwargs(ProcessingKwargs, total=False):
_defaults = {
"text_kwargs": {
"padding": False,
},
"videos_kwargs": {"return_metadata": True},
}


class Exaone4_5_Processor(Qwen2_5_VLProcessor):
tokenizer_class = "AutoTokenizer"

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nah, I see you added it in processing_auto, so now we can just delete these

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are we deleteing it?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I just delete Exaone4_5_Processor it breaks processing_exaone4_5.py

@zucchini-nlp zucchini-nlp Apr 29, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we delete the file processing.py entirely, It is mapped in auto_processing to qwen,ii think

Comment thread tests/models/exaone4_5/test_modeling_exaone4_5.py Outdated
@zucchini-nlp

Copy link
Copy Markdown
Member

You might also need a rebase, and at last run make fix-repo for repo consistency CI

@nuxlear

nuxlear commented Apr 27, 2026

Copy link
Copy Markdown
Contributor Author

@zucchini-nlp Sorry for the delay. I'm starting to address the feedback.

BTW, since EXAONE 4.5 was recently released in vLLM (v0.20.0), we need to keep some class names as-is for compatibility, including Exaone4_5_ImageProcessor.

Is there a recommended way to keep the model config unchanged while mapping the old name to the Transformers convention (e.g., Exaone4_5_ImageProcessorQwen2VLImageProcessor)?

If not, can I just patch this by aliasing the class name, e.g.,
from transformers import Qwen2VLImageProcessor as Exaone4_5_ImageProcessor in modeling_exaone4_5.py?

@zucchini-nlp

Copy link
Copy Markdown
Member

@nuxlear not sure I understand the vLLM part. We didn't yet release the model in transformers, so vLLM should be using its own integration without importing anything like from transformers import Exaone4_5_ImageProcessor

What part do we need to keep without changing?

@nuxlear

nuxlear commented Apr 27, 2026

Copy link
Copy Markdown
Contributor Author

Yes, we understand that it would be better for vLLM to use its own integration.

However, in this case, it would require updating our model config (e.g., exaone4_5_textexaone4, and Exaone4_5_[Image/Video]ProcessorQwen2VL[Image/Video]Processor) and submitting a separate PR to fix the imports in vLLM (see here).

So if there is a simple way to map or alias the existing config values and class names, we’d prefer that. Otherwise, we’d need to update the EXAONE 4.5 config, which would break compatibility with vLLM v0.20.0 and require additional changes.

@zucchini-nlp

Copy link
Copy Markdown
Member

@nuxlear so vllm is importing smth that doesn't yet exist and isn't released? 🫠

In that case, I don't think it falls into breaking BC since it wasn't released yet and there is nothing to break. Seems like vLLM can't support Exaone before transformers release anyway

@nuxlear

nuxlear commented Apr 27, 2026

Copy link
Copy Markdown
Contributor Author

@zucchini-nlp I understand. Then we will update the model config and open a patch PR later. I'll continue addressing the feedback 😃

@nuxlear

nuxlear commented Apr 29, 2026

Copy link
Copy Markdown
Contributor Author

@zucchini-nlp I think it's almost done, but I can't figure out why utils/check_repo.py is failing. It passes in my environment with the latest commit. (Edit: this was my bad. Never mind.)

To get the tests fully passing, we need to update the config from exaone4_5_text to exaone4, which may break the current vLLM job. Could you check the failing tests locally with this change? If it works on your side, we’ll proceed with updating the config and docs accordingly.

@nuxlear
nuxlear requested a review from zucchini-nlp April 29, 2026 05:58

@zucchini-nlp zucchini-nlp left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work, we can request a core maintainer review as last step before merging. I left some nitty-picky comments

Btw, are we deleting the unused processor class?

Comment thread docs/source/en/model_doc/exaone4_5.md Outdated
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForImageTextToText.from_pretrained(
model_id,
torch_dtype=torch.bfloat16,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: lets delete torch_dtype, we just merged a PR cleaning all docs. It default to dtype from config when loading

("depth_anything", "DepthAnythingConfig"),
("depth_pro", "DepthProConfig"),
("detr", "DetrConfig"),
("detr", "MaskFormerDetrConfig"),

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you revert it, looks like a bad rebase

self.num_key_value_groups = self.num_heads // self.num_key_value_heads
self.q_dim = self.num_heads * self.head_dim
self.kv_dim = self.num_key_value_heads * self.head_dim
self.qkv = nn.Linear(self.dim, self.q_dim + (self.kv_dim * 2), bias=True)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i was pointed recently, that we usually prefer unfused qkv, but I will leave it for core maintainer to decide. In any case, you won't have to change state-dicts, we will fuse-unfuse on the fly

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Honestly yes, but atp we have adopted too many qwen vl like checkpoints that have it fused. Usually it's easier on TP (unfused) but the vision backbones are small enough that it doesn't matter too much

self,
hidden_states: torch.Tensor,
cu_seqlens: torch.Tensor,
rotary_pos_emb: torch.Tensor | None = None,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: unused arg rotary_pos_emb (I realize it is copy from qwen but lets delete)

Comment on lines +279 to +280
elif position_ids.ndim > 2:
position_ids = position_ids[-1]

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this possible? I think we shouldn't allow it and let it error out naturally at some point

)


@auto_docstring(checkpoint="LGAI-EXAONE/EXAONE-4.5-33B")

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ultra nit: we don't need to add a ckpt everywhere, only in config classes :)

Comment on lines +520 to +530
class Exaone4_5_ProcessorKwargs(ProcessingKwargs, total=False):
_defaults = {
"text_kwargs": {
"padding": False,
},
"videos_kwargs": {"return_metadata": True},
}


class Exaone4_5_Processor(Qwen2_5_VLProcessor):
tokenizer_class = "AutoTokenizer"

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are we deleteing it?

@zucchini-nlp

zucchini-nlp commented Apr 29, 2026

Copy link
Copy Markdown
Member

To get the tests fully passing, we need to update the config from exaone4_5_text to exaone4, which may break the current vLLM job. Could you check the failing tests locally with this change? If it works on your side, we’ll proceed with updating the config and docs accordingly.

You mean the configs on the hub? Yes, looking at the tests looks like the changing model-type will fix it. If you cannot change config.json due to BC with vLLM/etc, I recomment to change it directly in code. For ex, smth like this inside a config.__post_init__:

# inlne comment here explaining why we override the model-type
if isinstance(text_config, dict):
    model_type = text_config.get('model_type', 'exaone4')
    if model_type = "exaone4_5_text": model_type = "exaone4"
    text_config = CONFIG_MAPPING[model_type](**text_cofnig)
elif text_config is None:
    text_config = CONFIG_MAPPING['exaone4']()

@nuxlear

nuxlear commented Apr 29, 2026

Copy link
Copy Markdown
Contributor Author

failed test: https://app.circleci.com/pipelines/github/huggingface/transformers/173444/workflows/6dec560d-7daf-4af6-bac6-fb9973c822b6/jobs/2292655
We will update the processor configs right after this PR is merged.

@HuggingFaceDocBuilderDev

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@github-actions

Copy link
Copy Markdown
Contributor

CI Results

Workflow Run ⚙️

Commit Info

Context Commit Description
RUN 66a4e642 workflow commit (merge commit)
PR e83132cb branch commit (from PR)
main 53b92b94 base commit (on main)

Model CI Report

2 new failed tests from this PR 😭

  • exaone4_5:
    tests/models/exaone4_5/test_modeling_exaone4_5.py::Exaone4_5_IntegrationTest::test_model_generation_image_text (✅ ⟹ ❌)
    tests/models/exaone4_5/test_modeling_exaone4_5.py::Exaone4_5_IntegrationTest::test_model_logits (✅ ⟹ ❌)

@vasqu

vasqu commented Apr 30, 2026

Copy link
Copy Markdown
Collaborator

I will take a look in a bit 🤗

@vasqu vasqu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks already very solid, left some comments but I think it's nothing major overall just smaller details

Comment thread docs/source/en/model_doc/exaone4_5.md
Comment thread src/transformers/models/auto/processing_auto.py
Comment thread src/transformers/models/exaone4_5/__init__.py
Comment thread src/transformers/models/exaone4_5/__init__.py Outdated
Comment thread src/transformers/models/exaone4_5/modular_exaone4_5.py Outdated
Comment thread src/transformers/models/exaone4_5/modular_exaone4_5.py
Comment thread src/transformers/models/exaone4_5/modular_exaone4_5.py Outdated
Comment thread tests/models/exaone4_5/test_modeling_exaone4_5.py Outdated
Comment thread tests/models/exaone4_5/test_modeling_exaone4_5.py Outdated
Comment thread tests/models/exaone4_5/test_modeling_exaone4_5.py
@nuxlear

nuxlear commented Apr 30, 2026

Copy link
Copy Markdown
Contributor Author

@vasqu Thanks for the review! Could you take a final look at the updates?

@nuxlear
nuxlear requested a review from vasqu April 30, 2026 20:53
@vasqu vasqu added the New model label May 4, 2026
@github-actions

github-actions Bot commented May 4, 2026

Copy link
Copy Markdown
Contributor

[For maintainers] Suggested jobs to run (before merge)

run-slow: auto, exaone4_5

@vasqu

vasqu commented May 4, 2026

Copy link
Copy Markdown
Collaborator

run-slow: exaone4_5

@vasqu vasqu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, I'm merging shortly just sanity checking with CI 🤗

@github-actions

github-actions Bot commented May 4, 2026

Copy link
Copy Markdown
Contributor

Workflow Run ⚙️

This comment contains run-slow, running the specified jobs:

models: ["models/exaone4_5"]
quantizations: []

@github-actions

github-actions Bot commented May 4, 2026

Copy link
Copy Markdown
Contributor

CI Results

Workflow Run ⚙️

Commit Info

Context Commit Description
RUN fe1b87e1 workflow commit (merge commit)
PR 8ff7a555 branch commit (from PR)
main 8c004ec6 base commit (on main)

⚠️ No test being reported (jobs are skipped or cancelled)!

@vasqu

vasqu commented May 4, 2026

Copy link
Copy Markdown
Collaborator

run-slow: exaone4_5

@github-actions

github-actions Bot commented May 4, 2026

Copy link
Copy Markdown
Contributor

Workflow Run ⚙️

This comment contains run-slow, running the specified jobs:

models: ["models/exaone4_5"]
quantizations: []

@github-actions

github-actions Bot commented May 4, 2026

Copy link
Copy Markdown
Contributor

CI Results

Workflow Run ⚙️

Commit Info

Context Commit Description
RUN 2697d317 workflow commit (merge commit)
PR 8fbf33bf branch commit (from PR)
main 8c004ec6 base commit (on main)

✅ No failing test specific to this PR 🎉 👏 !

@vasqu
vasqu enabled auto-merge May 4, 2026 13:07
@vasqu
vasqu added this pull request to the merge queue May 4, 2026
Merged via the queue into huggingface:main with commit a5b83a7 May 4, 2026
29 checks passed
@vasqu

vasqu commented May 4, 2026

Copy link
Copy Markdown
Collaborator

Nice, gz @nuxlear glad we got this in 🫡

@nuxlear

nuxlear commented May 4, 2026

Copy link
Copy Markdown
Contributor Author

Finally! Thank you @vasqu and @zucchini-nlp for your help and effort 😄

Exile333 pushed a commit to Exile333/transformers that referenced this pull request May 6, 2026
* Add EXAONE 4.5 modeling & config code

* Add test code for EXAONE 4.5 and Fix style

* Address PR feedback

* Address PR feedback

* Update EXAONE 4.5 docs

* Update docstring

* Address the feedback

* Fix docs

* Minor fix

* Minor fix

* Update docs

* Address the feedback

* Skip failed test

* Address the feedback

* Minor fix

* Revert skipping test

* Fix test

* Address the feedback

* fixup autos, nits, and logits test to CI

* fix date

* last quick fixes

---------

Co-authored-by: LG-AI-EXAONE <exaonemodels@lgresearch.ai>
Co-authored-by: vasqu <antonprogamer@gmail.com>
Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com>
banxingmjj pushed a commit to openanolis/DeepSpeed that referenced this pull request Jul 15, 2026
# Add EXAONE 4.5 model support for Inference V2

## Summary
Add support for LG AI Research's EXAONE 4.5 language model in DeepSpeed
Inference V2.

The EXAONE 4.5 text decoder shares EXAONE 4.0's post-norm + QK-Norm
parameter layout (`text_config.architectures ==
["Exaone4ForCausalLM"]`), so the transformer container and most of the
inference model are reused. EXAONE 4.5 is also a hybrid-attention model:
`sliding_attention` layers apply llama3-scaled RoPE, and
`full_attention` layers use no positional embedding (global NoPE). This
implementation follows that reference behavior.

## Changes
- New model implementation
`deepspeed/inference/v2/model_implementations/exaone4_5/`:
- `container.py`: non-transformer container for the multimodal
checkpoint layout. The LM weights are nested under
`model.language_model.` and `lm_head` stays top-level.
- `model.py`: hybrid attention on top of the reused EXAONE 4.0 forward.
- `sliding_attention` layers use trained-frequency RoPE with
llama3-scaled inverse frequencies. A unit test checks the values against
transformers' `ROPE_INIT_FUNCTIONS["llama3"]`.
- `full_attention` layers dispatch to a separate NoPE attention module.
- Sequence length is capped at `sliding_window` (4096). The dense
blocked attention kernel has no local mask, and at or below the window
size dense causal attention is equivalent to sliding attention. The cap
is enforced in the scheduler path (`get_kv_requirements`) and the direct
path (`maybe_allocate_kv`).
- `activation_dtype` handles the transformers v5 config (`dtype` vs
`torch_dtype`, str vs `torch.dtype`).
- `policy.py`: extracts `text_config`, reuses
`Exaone4TransformerContainer` under the `model.language_model.layers`
prefix, and leaves the vision tower (`model.visual.`) and the MTP head
(`mtp.`) unmapped.
- `exaone4/model.py`: extracted a behavior-neutral `_forward_attention`
seam for the per-layer dispatch, and fixed a latent aliasing crash in
`_apply_qk_norm`. For a single-row slice, `contiguous()` returns an
alias rather than a copy, so the write-back overlapped its own source
and every single-token decode step raised a RuntimeError. The q/k slices
are now cloned. This also affected EXAONE 4.0, which had not been
exercised on the decode path.
- `checkpoint/huggingface_engine.py`: derive `max_seq_length` from the
nested `text_config.max_position_embeddings` when the top-level
(multimodal) config doesn't have it.
- `kernels/ragged_ops/linear_blocked_kv_rotary`: the trained-RoPE path
now keeps the inverse frequencies and phase construction in FP32 instead
of coupling them to the Q/K activation dtype (repair by @tohtana, merged
from [Bias92#1](Bias92#1)).
Q/K/V and the KV cache stay FP16/BF16; the rotated outputs are cast back
to the activation dtype. A kernel test at position 4095 covers both
dtypes.
- Registered `exaone4_5` in `engine_factory.py` and
`model_implementations/__init__.py`.
- Unit tests for the llama3 frequency computation (against
transformers), the per-layer attention dispatch, and the sequence-length
cap.

## Why the mapping is exact
Checked against the real `LGAI-EXAONE/EXAONE-4.5-33B` checkpoint index
(1064 tensors). All 64 decoder layers expose exactly the 11 parameters
the 4.0 container maps, and every non-language-model tensor falls under
`model.visual.` (342) or `mtp.` (15), both declared unmapped.

## Testing
Validated on an A100 80GB (SXM4) at `73c325e`, CUDA 12.4, torch
2.4.1+cu124 / transformers 5.13.1, with the real
`LGAI-EXAONE/EXAONE-4.5-33B` checkpoint (bf16, shard SHA-256 verified):

- Unit tests for the diff-impacted scope, run on CI-matched pytest
8.3.5: rotary kernels 58/58, blocked attention 23/23, EXAONE 4.5 model
6/6.
- FP32-phase kernel test at position 4095 passes for both FP16 and BF16.
- Load: both shards load and every container initializes (mapping and
the `model.visual.`/`mtp.` skip both work). `max_sequence_length`
resolves to the enforced 4096 cap and the runtime rope buffer is
`torch.float32`.
- All pre-commit hooks (including clang-format on the CUDA files) pass
on the changed files.
- Greedy generation through `engine.put` (single-sequence decode, which
exercises the trained-rotary kernel, the NoPE module and the per-layer
dispatch):

```
'The capital of France is' -> ' Paris.  \nThe capital of Germany is Berlin.  \nThe capital of'
'대한민국의 수도는'          -> ' 어디인가요?\n서울입니다.\n질문은 대한민국의 수도를 묻고 있으며...'
'2 + 2 ='                 -> ' 4.\n\nBut in the example, they said "the sum of all elements is'
'The opposite of hot is'  -> ' cold.  \nThe opposite of happy is sad.  \nThe opposite of'
```

Note: the direct-put validation used `do_checks=False` because
`engine.can_schedule` currently passes `state_manager.free_blocks` (a
Python `list`, despite the `torch.Tensor` annotation) into
`get_kv_requirements`, which expects an int. That is a pre-existing type
mismatch that affects all models on the `put(do_checks=True)` path. MII
drives the engine via `query()` + `put(do_checks=False)`, so it never
hits it. Happy to file that separately.

## Scope
Serves the base autoregressive text path of EXAONE 4.5 (e.g.
`LGAI-EXAONE/EXAONE-4.5-33B`, bf16) up to 4096 tokens.

Out of scope / follow-ups:
- Vision tower (VLM, `Exaone4_5ForConditionalGeneration`)
- MTP self-speculative decoding
- FP8 / AWQ quantized variants (only fp16/bf16, same as EXAONE 4.0)
- Long context (>4096), which needs a local attention mask in the dense
blocked attention kernel

The merged EXAONE 4.0 implementation has the same uniform-RoPE issue on
its `full_attention` layers (plus unscaled llama3 rope). This PR keeps
4.0 behavior unchanged apart from the decode crash fix. I'd like to fix
4.0 in a follow-up PR so the two changes stay independently reviewable.

## Requirements
`transformers >= 5.3.0`. EXAONE 4.5 landed in transformers 5.3
(huggingface/transformers#45471).

---------

Signed-off-by: Bias92 <pewpewplay315@gmail.com>
Signed-off-by: Masahiro Tanaka <mtanaka@anyscale.com>
Co-authored-by: Masahiro Tanaka <mtanaka@anyscale.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants