Modular playground

[itazap]

Modular Detector Updates:

Hub / index: FAISS index. The new index is at: itazap/transformers_code_embeddings_v3 (old: hf-internal-testing/transformers_code_embeddings)

Model: Qwen/Qwen3-Embedding-4B, now using last-token pooling instead of mean pooling

Sanitization: type hints, dtype save/cast boilerplate, and minor constructor kwargs (e.g. bias=) are stripped
Inheritance modular search: parse all modular_*.py imports to build themodel ancestry graph, then walks ancestor chains to parent-model classes that might not land in the raw top-k (e.g. a query matching Qwen3MLP will also score LlamaMLP via the Qwen3→Llama modular inheritance chain).

Time filter: models released after the query model are excluded (uses doc dates from docs/source/en/model_doc/*.md).

Output / print changes:

Model match summary: ranked table of candidate models with num_matched × mean_score² composite score, redundant ancestors filtered out (aka self-contained)
Suggested modular inheritance: greedy set-cover picks ≤3 parent models and assigns each class to its best parent, with --generate-prompt for an agent (WIP)

new cli flags
--ignore-models  # exclude specific models
--summary-only        # skip per-symbol tables
--generate-prompt [file]   # write an agent prompt for modular file creation
--push-only          # push index to Hub without rebuilding
--use_jaccard / --no-use_jaccard # now on by default

Evaluation:

utils/run_modular_detector_eval.py: evaluates detector accuracy against ground-truth modular inheritance.
Eval dataset: itazap/modeling-dataset - model_name, original_modeling_code (not available for all, but found via private GitHub repos, first commits, etc.), current_modular_code, bases (models inherited from in modular)
Runs: modular_model_detector on every model's “original” modeling code, checks whether the ground-truth parent models appear in the detector's top-1/3/5 suggestions, and writes per-model results to JSON.

Model subsets:
• FILTERED_MODELS - the ~30 models currently used for eval (the ones I could find original code for)
• LIGHT_MODELS - small subset of <=15 classes for quick testing (--light flag)

CLI flags:
--eval-source hub-original|json
--eval-hub-repo     # defaults to itazap/modeling-dataset
--output results.json
--light         # restrict to LIGHT_MODELS
--limit N        # first N entries only
--workers N       # parallel threads (OOM risk with >1)
--clear-cache-each-run  # on by default
--reload-analyzer-each-run # recreate analyzer per entry to free GPU memory (edited)

Modular Inheritance - Eval Dataset

Naiive but a dataset mapping our models to the model class(es) they inherit form in their modular file

https://huggingface.co/datasets/itazap/modular-model-eval/viewer

Usage:


python utils/modular_model_detector.py --modeling-file src/transformers/models/sarvam/modeling_sarvam.py 
------------




Model class match summary

Total classes: 11

Models with most matched classes:
Model            | Matched | Pct   | Mean score | Classes                                                                                                           
-----------------+---------+-------+------------+-------------------------------------------------------------------------------------------------------------------
deepseek_v2      | 6/11    | 54.5% | 0.8894     | MoEGate, SarvamMLAAttention, SarvamMLADecoderLayer, SarvamMLAMLP, SarvamMLAMoE, SarvamMLAModel                    
ernie4_5_vl_moe  | 6/11    | 54.5% | 0.7658     | MoEGate, SarvamMLADecoderLayer, SarvamMLAMLP, SarvamMLAMoE, SarvamMLARotaryEmbedding, SarvamMLAYarnRotaryEmbedding
qwen3_omni_moe   | 5/11    | 45.5% | 0.7086     | SarvamMLADecoderLayer, SarvamMLAMLP, SarvamMLARMSNorm, SarvamMLARotaryEmbedding, SarvamMLAYarnRotaryEmbedding     




run_modular_detector_eval.py output:

=== Eval summary (models that have bases and a non-empty detector summary) ===
Total with labels and summary: 29
Top-1 accuracy (first suggested model in bases): 20.69% (6/29)
Top-3 accuracy (any base in top 3): 51.72% (15/29)
Top-5 accuracy (any base in top 5): 65.52% (19/29)
Total eval entries with bases: 29 (skipped/errors: 0)

=== Per-model predictions ===
model                bases                                                predicted top 3                                
-------------------  ---------------------------------------------------  -------------------------------------------
biogpt               bart,opt                                             trocr, whisper, patchtst
camembert            roberta                                              xmod, xlm_roberta_xl, xlm_roberta
conditional_detr     deformable_detr,detr                                 table_transformer, detr, dab_detr
deepseek_v2          llama,qwen2_moe                                      llama, nemotron, deepseek_v3
deepseek_v3          llama,mixtral,qwen2_moe                              nemotron, llama, glm4_moe
deformable_detr      detr                                                 grounding_dino, detr, conditional_detr
falcon_mamba         mamba                                                mamba
gpt_neox             llama                                                gptj, bigbird_pegasus, openai
granite              llama                                                diffllama, nemotron, moshi
granitemoe           granite,jetmoe,llama,mixtral                         granitemoehybrid, mixtral, granitemoeshared
hubert               wav2vec2                                             mbart, plbart, mt5
hunyuan_v1_moe       hunyuan_v1_dense,llama,mixtral                       hunyuan_v1_dense, qwen2_vl, llama
jetmoe               llama,mixtral                                        moshi, qwen2_vl, nemotron
mistral              llama                                                phi3, clvp, llama
olmo                 llama                                                nemotron, cohere, diffllama
olmoe                gemma,llama,mixtral,qwen2_moe                        flex_olmo, llama, mixtral
paddleocr_vl         ernie4_5,qwen2_5_omni,qwen2_vl,siglip,video_llama_3  llama, arcee, gemma
persimmon            llama                                                stablelm, nemotron, llama
phi                  clip,llama                                           auto, bart, esm
phi3                 mistral,phi                                          llama, moshi, nemotron
phimoe               llama,mixtral                                        nemotron, mixtral, flex_olmo
qwen2                gemma2,llama,mistral                                 nemotron, qwen2_vl, llama
qwen2_moe            gemma,gemma2,llama,mixtral                           qwen3_moe, nemotron, qwen2_vl
sew                  wav2vec2                                             hubert
switch_transformers  t5                                                   longt5, udop, umt5
unispeech            wav2vec2                                             wav2vec2, unispeech_sat, longformer
unispeech_sat        wav2vec2                                             wav2vec2, unispeech, longformer
wavlm                wav2vec2                                             wav2vec2, longformer, xlnet
xlm_roberta          roberta                                              camembert, xlm_roberta_xl, xmod

UPDATE:

=== Eval summary (models that have bases and a non-empty detector summary) ===
Total with labels and summary: 34
Top-1 accuracy (first suggested model in bases): 76.47% (26/34)
Top-3 accuracy (any base in top 3): 97.06% (33/34)
Top-5 accuracy (any base in top 5): 100.00% (34/34)
Total eval entries with bases: 34 (skipped/errors: 0)

=== Per-model predictions ===
model                bases                                                predicted                               
-------------------  ---------------------------------------------------  ----------------------------------------
biogpt               bart,opt                                             bart, opt, trocr
camembert            roberta                                              roberta, bert, xmod
conditional_detr     deformable_detr,detr                                 detr, table_transformer, deformable_detr
deepseek_v2          llama,qwen2_moe                                      llama, gemma, mistral
deepseek_v3          llama,mixtral,qwen2_moe                              llama, gemma, mistral
deformable_detr      detr                                                 grounding_dino, detr, conditional_detr
ernie4_5             glm,llama,olmo                                       llama, mistral, glm
ernie4_5_moe         ernie4_5,llama,mixtral,qwen3_moe                     qwen2, qwen3_moe, gemma
falcon_mamba         mamba                                                mamba, mamba2, falcon_h1
gpt_neox             llama                                                llama, gpt_neox_japanese, gptj
granite              llama                                                llama, mistral, diffllama
granitemoe           granite,jetmoe,llama,mixtral                         jetmoe, granitemoeshared, granite
hubert               wav2vec2                                             unispeech_sat, unispeech, wav2vec2
hunyuan_v1_moe       hunyuan_v1_dense,llama,mixtral                       hunyuan_v1_dense, mistral, mixtral
jetmoe               llama,mixtral                                        mixtral, llama, granitemoe
mistral              llama                                                llama, phi3, gemma
olmo                 llama                                                llama, mistral, nemotron
olmoe                gemma,llama,mixtral,qwen2_moe                        qwen2_moe, mistral, flex_olmo
paddleocr_vl         ernie4_5,qwen2_5_omni,qwen2_vl,siglip,video_llama_3  qwen2_vl, ernie4_5, llama
persimmon            llama                                                llama, stablelm, nemotron
phi                  clip,llama                                           llama, stablelm, persimmon
phi3                 mistral,phi                                          llama, mistral, phi
phimoe               llama,mixtral                                        mixtral, mistral, llama
qwen2                gemma2,llama,mistral                                 llama, mistral, gemma
qwen2_moe            gemma,gemma2,llama,mixtral                           mixtral, mistral, llama
qwen3_5              qwen3_next,qwen3_vl                                  qwen3_vl, qwen2_vl, qwen2_5_vl
qwen3_5_moe          qwen3_5,qwen3_next,qwen3_vl_moe                      qwen3_next, jamba, qwen2
qwen3_omni_moe       qwen3,qwen3_moe,qwen3_vl_moe                         qwen2_5_omni, qwen3_vl_moe, qwen2_vl
sew                  wav2vec2                                             hubert, wav2vec2, unispeech_sat
switch_transformers  t5                                                   longt5, udop, t5
unispeech            wav2vec2                                             wav2vec2, unispeech_sat, wavlm
unispeech_sat        wav2vec2                                             wav2vec2, unispeech, wavlm
wavlm                wav2vec2                                             wav2vec2, unispeech, unispeech_sat
xlm_roberta          roberta                                              camembert, xlm_roberta_xl, xmod
(uvenv) ita_zaporozhets@ip-26-0-162-14:/fsx/ita_zaporozhets/transformers$ 

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[itazap]

run-slow: persimmon

[github-actions]

[For maintainers] Suggested jobs to run (before merge)

run-slow: persimmon