Dynamic auto mapping

[zucchini-nlp]

What does this PR do?

As per title, deletes long lines of auto-mapping and dynamically builds it from code.

The whole auto-generation runs with AST and takes ~5 sec on my machine, so we don't infer anything at run-time. Instead it is built once and saved, similar to modular files. This also uncovered a few issues, e.g. the same model_type is used with different config classes, so imo it is a good opportunity to be more strict about model naming. The check_auto will now complain if there a not one-to-one mapping in configs, because it might lead to incorrect AutoXXX loading

I checked the mapping content that we're not missing any entry from main with this script. Nothing is missing except for a couple keys I deleted on purpose. Those just do not exist, in code or in official hub ckpt.

for name, value in CORRECT_CONFIG_MAPPING_NAMES.items():
    if CONFIG_MAPPING_NAMES.get(name, "none") != value:
        print(f"{name}: {value} != {CONFIG_MAPPING_NAMES.get(name, 'none')}")

print("\n SPECIAL_MODEL_TYPE_TO_MODULE_NAME")
for name, value in CORRECT_SPECIAL_MODEL_TYPE_TO_MODULE_NAME.items():
    if SPECIAL_MODEL_TYPE_TO_MODULE_NAME.get(name, "none") != value:
        print(f"{name}: {value} != {SPECIAL_MODEL_TYPE_TO_MODULE_NAME.get(name, 'none')}")

print("\n IMAGE_PROCESSOR_MAPPING_NAMES")
for name, value in CORRECT_IMAGE_PROCESSOR_MAPPING_NAMES.items():
    if IMAGE_PROCESSOR_MAPPING_NAMES.get(name, "none") != value:
        print(f"{name}: {value} != {IMAGE_PROCESSOR_MAPPING_NAMES.get(name, 'none')}")

print("\n VIDEO_PROCESSOR_MAPPING_NAMES")
for name, value in CORRECT_VIDEO_PROCESSOR_MAPPING_NAMES.items():
    if VIDEO_PROCESSOR_MAPPING_NAMES.get(name, "none") != value:
        print(f"{name}: {value} != {VIDEO_PROCESSOR_MAPPING_NAMES.get(name, 'none')}")

NOTE: MODEL_MAPPING_NAMES is deleted entirely, as it is not used anywhere in the codebase. It holds official non-lowercase names for each model and we don't use those anywhere, only keys

[zucchini-nlp]

run-slow: eurobert, nougat, musicflamingo

[github-actions]

Workflow Run ⚙️

This comment contains run-slow, running the specified jobs:

models: ["models/eurobert", "models/musicflamingo", "models/nougat"]
quantizations: []

[github-actions]

CI Results

Workflow Run ⚙️

Commit Info

Context	Commit	Description
RUN	1ccf7332	workflow commit (merge commit)
PR	7b26de04	branch commit (from PR)
main	5b565a58	base commit (on main)

✅ No failing test specific to this PR 🎉 👏 !

[zucchini-nlp]

run-slow: bert, gpt2, t5, modernbert, vit, clip, detr, table_transformer, got_ocr2, whisper, wav2vec2, qwen2_audio, speech_t5, csm, llama, gemma3, qwen2, mistral3, qwen2_5_vl, llava, smolvlm, internvl, gemma3n, gpt_oss, qwen2_5_omni, pi0

[github-actions]

Workflow Run ⚙️

This comment contains run-slow, running the specified jobs:

models: ["models/bert", "models/clip", "models/csm", "models/detr", "models/gemma3", "models/gemma3n", "models/got_ocr2", "models/gpt2", "models/gpt_oss", "models/internvl", "models/llama", "models/llava", "models/mistral3", "models/modernbert", "models/pi0", "models/qwen2", "models/qwen2_5_omni", "models/qwen2_5_vl", "models/qwen2_audio", "models/smolvlm", "models/t5", "models/table_transformer", "models/vit", "models/wav2vec2", "models/whisper"]
quantizations: []

[github-actions]

CI Results

Workflow Run ⚙️

Commit Info

Context	Commit	Description
RUN	b4c868bf	workflow commit (merge commit)
PR	bdba64aa	branch commit (from PR)
main	b6f9463e	base commit (on main)

Model CI Report

❌ 11 new failed tests from this PR 😭

• gemma3:
  tests/models/gemma3/test_modeling_gemma3.py::Gemma3IntegrationTest::test_model_4b_batch (❌ ⟹ ❌)
  tests/models/gemma3/test_modeling_gemma3.py::Gemma3IntegrationTest::test_model_4b_batch_crops (❌ ⟹ ❌)
  tests/models/gemma3/test_modeling_gemma3.py::Gemma3IntegrationTest::test_model_4b_bf16 (❌ ⟹ ❌)
  tests/models/gemma3/test_modeling_gemma3.py::Gemma3IntegrationTest::test_model_4b_crops (❌ ⟹ ❌)
  tests/models/gemma3/test_modeling_gemma3.py::Gemma3IntegrationTest::test_model_4b_multiimage (❌ ⟹ ❌)

• llava:
  tests/models/llava/test_modeling_llava.py::LlavaForConditionalGenerationIntegrationTest::test_batched_generation (❌ ⟹ ❌)
  tests/models/llava/test_modeling_llava.py::LlavaForConditionalGenerationIntegrationTest::test_generation_siglip_backbone (❌ ⟹ ❌)
  tests/models/llava/test_modeling_llava.py::LlavaForConditionalGenerationIntegrationTest::test_pixtral (❌ ⟹ ❌)

• pi0:
  tests/models/pi0/test_modeling_pi0.py::PI0ModelIntegrationTest::test_pi0_base_libero (❌ ⟹ ❌)
  tests/models/pi0/test_modeling_pi0.py::PI0ModelIntegrationTest::test_pi0_base_reference_values (❌ ⟹ ❌)
  tests/models/pi0/test_modeling_pi0.py::PI0ModelIntegrationTest::test_train_pi0_base_libero (❌ ⟹ ❌)

[zucchini-nlp]

I dont know why it shows these models as new failures, the error message is identical to main branch

[github-actions]

[For maintainers] Suggested jobs to run (before merge)

run-slow: auto, eurobert, fastspeech2_conformer, idefics, idefics2, kosmos2, kosmos2_5, lighton_ocr

[zucchini-nlp]

run-slow: eurobert, fastspeech2_conformer, idefics, idefics2, kosmos2, kosmos2_5, lighton_ocr, code_llama, nougat, vision_encoder_decoder

[github-actions]

Workflow Run ⚙️

This comment contains run-slow, running the specified jobs:

models: ["models/code_llama", "models/eurobert", "models/fastspeech2_conformer", "models/idefics", "models/idefics2", "models/kosmos2", "models/kosmos2_5", "models/lighton_ocr", "models/nougat", "models/vision_encoder_decoder"]
quantizations: []

[github-actions]

CI Results

Workflow Run ⚙️

Commit Info

Context	Commit	Description
RUN	bab8a383	workflow commit (merge commit)
PR	c5bc6a3d	branch commit (from PR)
main	8426e7e6	base commit (on main)

Model CI Report

❌ 2 new failed tests from this PR 😭

• vision_encoder_decoder:
  tests/models/vision_encoder_decoder/test_modeling_vision_encoder_decoder.py::NougatModelIntegrationTest::test_forward_pass (❌ ⟹ ❌)
  tests/models/vision_encoder_decoder/test_modeling_vision_encoder_decoder.py::NougatModelIntegrationTest::test_generation (❌ ⟹ ❌)

[zucchini-nlp]

Also not related and failing on main, the repo is not found in hub!

Then merging it, important models tests are passing and the keys from mapping aren't lost on the way. Instead new entries are added for all sub-configs, so now any sub-backbone can be loaded with Auto classes as long as we add proper keys in modeling_auto.py

[tomaarsen]

Quick question: in Sentence Transformers I'm using MODEL_MAPPING_NAMES to easily get all keys of architectures in transformers for some tests (and only in the tests, so the changes are not breaking for me or anything like that). Should I move to CONFIG_MAPPING_NAMES.keys() instead from the next version?

• Tom Aarsen

[zucchini-nlp]

@tomaarsen yes, the keys in CONFIG_MAPPING_NAMES will do the job as well :)