[Feat] Adding Intern-S1

[hhaAndroid]

Adding Intern-S1

This PR adds the support of codes for the Intern-S1 models. Please visit https://huggingface.co/internlm/Intern-S1

Features

• Strong performance across language and vision reasoning benchmarks, especially scientific tasks.
• Continuously pretrained on a massive 5T token dataset, with over 50% specialized scientific data, embedding deep domain expertise.
• Dynamic tokenizer enables native understanding of molecular formulas, protein sequences, and seismic signals.

Usage

from transformers import AutoProcessor, AutoModelForImageTextToText
import torch

model_checkpoint = 'xxxx'
processor = AutoProcessor.from_pretrained(model_checkpoint)
model = AutoModelForImageTextToText.from_pretrained(model_checkpoint, device_map="auto", torch_dtype="auto")
messages = [
        {
            "role": "user",
            "content": [
                {"type": "image",
                 "url": "http://images.cocodataset.org/val2017/000000039769.jpg"},
                {"type": "text", "text": "Please describe the image shortly."},
            ],
        }
    ]

inputs = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=True, return_dict=True,
                                           return_tensors="pt").to(model.device, dtype=torch.bfloat16)

generate_ids = model.generate(**inputs, max_new_tokens=32768)
decoded_output = processor.decode(generate_ids[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True)
print(decoded_output)

Progress

• add modeling py
• add tokenizer.py
• add test
• fix lint

[Rocketknight1]

cc @zucchini-nlp for VLMs!

[zucchini-nlp]

Taking a look tomorrow

[zucchini-nlp]

Hey, sorry for late review, got caught up in another model release.

The model looks very much like InternVL and I want us to re-use as much code as possible with modular. In long term it will make maintenance easier for us, and much much faster review process if we can spot the differences between models. I left comments below about which class can be re-used from where

Feel free to tag me when it is ready for re-review or if you need any assistance :)

[hhaAndroid]

@zucchini-nlp Hello, I've revised a new version as requested. However, regarding the usage of can_record_outputs, I'm unsure if it fits my scenario. After adapting it for the MoE model, I need to pass the output_router_logits parameter to the MoE LLM, rather than just capturing the output results. Looking forward to your next round of review comments.

[zucchini-nlp]

Super clean after using the modular, thanks for iterating! There are some bits left especially moving all code into modular. We shouldn't be importing from other models unless it is in the modular file :)

Also, I will take a look at the can_record_output thing this week, would be nice to get it sorted

Update: Oh btw, let's make CI green and fix failing tests. You might need to rebase if unrelated test are failing

[hhaAndroid]

@zucchini-nlp Hello, I have made a round of fixes based on your comments and also left some issues I encountered. Could you please take a look when you have time to see if it is reasonable now? Regarding the test CI issues, I will fix them all together after the review structure is confirmed to be fine.

[zucchini-nlp]

Btw, the failing compile test can be skipped if the LM backbone is MoE. We don't have fullgraph compilation supported with MoE yet

[zucchini-nlp]

LGTM, replied to you question above and we will ask core maintainer's review.

Totally forgot that we want to delete self.is_moe_model = False, taking a look at can_return_outputs now 👀

[hhaAndroid]

@zucchini-nlp Hello, I have fixed the CI, but there are three issues caused by numerical precision. I am not sure how to fix them or whether they need to be fixed.

FAILED tests/models/interns1/test_modeling_interns1.py::InternS1ModelTest::test_eager_matches_sdpa_inference_08_fp32_pad_left_sdpa_kernels - ValueError: mean relative difference for hidden_states: 4.503e-05, torch atol = 1e-06, torch rtol = 0.0001
FAILED tests/models/interns1/test_modeling_interns1.py::InternS1ModelTest::test_eager_matches_sdpa_inference_12_fp32_pad_right_sdpa_kernels - ValueError: mean relative difference for hidden_states: 4.313e-05, torch atol = 1e-06, torch rtol = 0.0001
FAILED tests/models/interns1/test_modeling_interns1.py::InternS1ModelTest::test_left_padding_compatibility - AssertionError: Tensor-likes are not close!
``

[zucchini-nlp]

Can you check tests/models/interns1/test_modeling_interns1.py::InternS1ModelTest::test_left_padding_compatibility if the difference is high?

[hhaAndroid]

Can you check tests/models/interns1/test_modeling_interns1.py::InternS1ModelTest::test_left_padding_compatibility if the difference is high?

The interns1 architecture and code are very similar to internvl. I noticed that the currently merged internvl-related CI is also failing.

[zucchini-nlp]

Oke, feel free to skip them in that case with unitest.skip(reason=) or overwrite if needed. These tests are probably slow CI and aren't run under each PR so we didn't notice failures

Let me just trigger first all tests and check if they fail in runners

[zucchini-nlp]

run-slow: interns1

[hhaAndroid]

Oke, feel free to skip them in that case with unitest.skip(reason=) or overwrite if needed. These tests are probably slow CI and aren't run under each PR so we didn't notice failures

Let me just trigger first all tests and check if they fail in runners

Okay, looking forward to your results.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[zucchini-nlp]

run-slow: interns1

[github-actions]

This comment contains run-slow, running the specified jobs:

models: ['models/interns1']
quantizations: [] ...

[zucchini-nlp]

I will trigger slow tests, can you not commit anything until it is finished running pls

[zucchini-nlp]

run-slow: interns1

[github-actions]

This comment contains run-slow, running the specified jobs:

models: ['models/interns1']
quantizations: [] ...

[zucchini-nlp]

Tests are fine, only one failed due to OOM

FAILED tests/models/interns1/test_modeling_interns1.py::InternS1ModelTest::test_flex_attention_with_grads - torch._inductor.exc.InductorError: RuntimeError: No valid triton configs. OutOfMemoryError: out of resource: triton_tem_fused_0 Required: 106496 Hardware limit:101376 Reducing block sizes or `num_stages` may help.

[hhaAndroid]

@zucchini-nlp Thank you. Waiting for the results of other reviewers.

[github-actions]

[For maintainers] Suggested jobs to run (before merge)

run-slow: auto, intern_s1

[github-actions]

[For maintainers] Suggested jobs to run (before merge)

run-slow: auto, fast_vlm, intern_s1

[github-actions]

[For maintainers] Suggested jobs to run (before merge)

run-slow: auto, intern_s1

[github-actions]

[For maintainers] Suggested jobs to run (before merge)

run-slow: auto, intern_s1

[Anti-Entrophic]

The code is ready for review now. Please let us know your suggestions, thank you. @Cyrilvallez @ArthurZucker

[Cyrilvallez]

cc @ArthurZucker for tokenizers again after the refactor, not sure exactly how it works now?

[github-actions]

[For maintainers] Suggested jobs to run (before merge)

run-slow: auto, intern_s1