Support having multiple sub-processors (of any kind) in the same processor by yonigozlan · Pull Request #42667 · huggingface/transformers

yonigozlan · 2025-12-05T21:43:43Z

What does this PR do?

Also standardizes the few remaining processors left to standardize.

yonigozlan · 2025-12-05T21:45:13Z

-    attributes = ["feature_extractor", "tokenizer"]
-    feature_extractor_class = "WhisperFeatureExtractor"
-    tokenizer_class = "Qwen2TokenizerFast"
-


Cc @eustlb , moved this to the auto files to have one source of truth, and removed attributes as they are now auto-detected

yonigozlan · 2025-12-05T21:47:31Z



 class LasrProcessor(ProcessorMixin):
-    tokenizer_class = "ParakeetTokenizerFast"


Cc @eustlb Same here, although is this supposed to be Parakeet or lasr tokenizer by default?

HuggingFaceDocBuilderDev · 2025-12-05T22:00:53Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

zucchini-nlp

Native-support for several processors of the same type is a great feature to have! I am not sure though if the current auto-behavior will work well, it is likely to raise issues with power users who don't follow the "patterns" from this PR

We could handle it gracefully when no config found or names don't match instead of raising an error, WDYT?

zucchini-nlp · 2025-12-08T10:07:25Z

+def _get_modality_for_attribute(attribute_name: str) -> str:
+    """
+    Get the canonical modality type for a given attribute name.
+
+    For example:
+    - "image_processor" -> "image_processor"
+    - "encoder_image_processor" -> "image_processor"
+    - "text_tokenizer" -> "tokenizer"
+    - "my_feature_extractor" -> "feature_extractor"
+    """
+    for modality in MODALITY_TO_AUTOPROCESSOR_MAPPING.keys():
+        if modality in attribute_name:
+            return modality
+    raise ValueError(
+        f"Cannot determine modality for attribute '{attribute_name}'. "
+        f"Attribute name must contain one of: {list(MODALITY_TO_AUTOPROCESSOR_MAPPING.keys())}"
+    )


imo this is simplified a lot. Users do not always call attributes following the pattern and also might want to use their own processing classes. There is a lot of inheritance and patching in custom code afaik, which can't be simplified to _get_modality_for_attribute imo

_get_modality_for_attribute is called on "attributes" names obtain with cls.get_attributes(), which are already filtered "attributes" corresponding to the sub-processors. So I think this should be fine, unless we want users being able to define sub-processors which variable name don't contain the sub-processor type, but I don't know why we would want that, and how we would detect what kind of sup-processor they are in that case.

I guess attributes is a misnomer here really, and we should maybe change it here to subprocessors, but it was named this before the refactor so I didn't want to change it to not break bc. It might be wort changing it though as it's causing a lot of confusion

+1, the naming is indeed causing confusion. Probably I am thinking about too niche cases, as I haven't personally seen many power users using a custom processor. I just realized that were always strict with processor args/kwargs at initialization time 😄

zucchini-nlp · 2025-12-08T10:14:52Z

+            else:
+                raise ValueError(
+                    f"Cannot find config for {sub_processor_type} in processor_config.json. "
+                    f"Available keys: {list(processor_dict.keys())}"
+                )


also not sure it's a good idea to raise an error if the attribute has no config dict. One possible use-case is when a processor has optional attributes that are not available on purpose (see #40447)

Not sure I understood the use case in the issue linked :(, do you have an example? This code path would only be used by processor_dict that corresponds to a sub-processor

same as above, nvm. It's not breaking BC and I was thinking about a new feat which we don't yet have

zucchini-nlp

Oke, since functionality-wise we are doing the same as before and I don't see possible edge cases, approving the PR! Great job, multiple subprocessors will be super helpful for multimodal models

Cyrilvallez

Very nice!

Cyrilvallez · 2025-12-11T16:30:57Z

+            if "tokenizer" in sub_processor_type:
+                tokenizer = cls._load_tokenizer_from_pretrained(
+                    sub_processor_type, pretrained_model_name_or_path, subfolder=subfolder, **kwargs
                )
                args.append(tokenizer)


Why do we need different code paths for tokenizer and other processor types?

Because additional tokenizers will be saved in subfolders (as they have their own files that need to be saved), while other additional subprocessors will be saved directly in processor_config.json

Ha indeed, other processors do not need to save other files! Thanks for the explanation!

itazap · 2025-12-11T16:46:55Z

        ("align", "BertTokenizer" if is_tokenizers_available() else None),
        ("arcee", "LlamaTokenizer" if is_tokenizers_available() else None),
        ("aria", "LlamaTokenizer" if is_tokenizers_available() else None),
+        ("audioflamingo3", "Qwen2TokenizerFast" if is_tokenizers_available() else None),


nit but for v5 we drop the "Fast", we can update this to Qwen2Tokenizer ! == the Fast one 😊

(same for updating ParakeetTokenizerFast to drop the "fast" (from file name too)

Looks like Parakeet tokenizer is not yet standardized to use TokenizersBackend :/, and still uses the name ParakeetTokenizerFast

…n/transformers into support-multi-sub-proc

github-actions · 2025-12-16T21:42:08Z

[For maintainers] Suggested jobs to run (before merge)

run-slow: audioflamingo3, auto, fuyu, lasr, phi4_multimodal, pix2struct, pixtral

github-actions · 2025-12-16T21:49:15Z

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=42667&sha=1c7335

…essor (huggingface#42667) * support saving/loading multiple sub_processor of the same kind * standardize all processors * remove tokenizer_class from lasr * fix modular * fix kwargs logic * override _load_tokenizer_from_pretrained in pixtral and fuyu * fix missing subfolder loading primary tokenizer * remove Fast suffix in tokenization auto * revert to ParakeetTokenizerFast (not yet updated) * Set TestMistralCommonBackend as flaky

yonigozlan added 2 commits December 5, 2025 21:23

support saving/loading multiple sub_processor of the same kind

9c6357c

standardize all processors

f3bd01c

yonigozlan requested review from ArthurZucker, Cyrilvallez and zucchini-nlp December 5, 2025 21:43

yonigozlan commented Dec 5, 2025

View reviewed changes

remove tokenizer_class from lasr

c84b564

yonigozlan commented Dec 5, 2025

View reviewed changes

Comment thread src/transformers/models/pix2struct/processing_pix2struct.py

fix modular

abd038d

fix kwargs logic

114a48b

yonigozlan mentioned this pull request Dec 5, 2025

More robust processor from pretrained #42668

Open

zucchini-nlp reviewed Dec 8, 2025

View reviewed changes

zucchini-nlp approved these changes Dec 9, 2025

View reviewed changes

Merge remote-tracking branch 'upstream/main' into support-multi-sub-proc

fde1c81

yonigozlan force-pushed the support-multi-sub-proc branch from 51fc687 to fde1c81 Compare December 9, 2025 16:46

yonigozlan and others added 3 commits December 9, 2025 17:08

override _load_tokenizer_from_pretrained in pixtral and fuyu

ed400c3

fix missing subfolder loading primary tokenizer

7c79ebe

Merge branch 'main' into support-multi-sub-proc

81a50cc

Cyrilvallez approved these changes Dec 11, 2025

View reviewed changes

itazap reviewed Dec 11, 2025

View reviewed changes

yonigozlan added 2 commits December 11, 2025 17:55

remove Fast suffix in tokenization auto

9593409

Merge branch 'support-multi-sub-proc' of https://github.com/yonigozla…

c8a4b87

…n/transformers into support-multi-sub-proc

yonigozlan enabled auto-merge (squash) December 11, 2025 17:56

yonigozlan disabled auto-merge December 11, 2025 17:58

yonigozlan and others added 4 commits December 11, 2025 18:02

revert to ParakeetTokenizerFast (not yet updated)

feccf7c

Merge branch 'main' into support-multi-sub-proc

5037ba9

Merge branch 'main' into support-multi-sub-proc

84832cc

Merge branch 'main' into support-multi-sub-proc

60a6bf0

yonigozlan and others added 5 commits December 15, 2025 11:02

Merge branch 'main' into support-multi-sub-proc

085e2a8

Merge branch 'main' into support-multi-sub-proc

c9fd0ce

Merge remote-tracking branch 'upstream/main' into support-multi-sub-proc

51c35cb

Merge branch 'support-multi-sub-proc' of https://github.com/yonigozla…

e88bb3b

…n/transformers into support-multi-sub-proc

Set TestMistralCommonBackend as flaky

1c73351

yonigozlan merged commit dd24a80 into huggingface:main Dec 16, 2025
25 checks passed

yonigozlan mentioned this pull request Dec 16, 2025

multiple tokenizers with different filenames can save now #41837

Closed



		class LasrProcessor(ProcessorMixin):
		tokenizer_class = "ParakeetTokenizerFast"

Uh oh!

Conversation

yonigozlan commented Dec 5, 2025

What does this PR do?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

HuggingFaceDocBuilderDev commented Dec 5, 2025

Uh oh!

zucchini-nlp left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yonigozlan Dec 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zucchini-nlp left a comment

Choose a reason for hiding this comment

Uh oh!

Cyrilvallez left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Dec 16, 2025

Uh oh!

github-actions Bot commented Dec 16, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

yonigozlan Dec 8, 2025 •

edited

Loading