Support Audio Flamingo Next checkpoints by lashahub · Pull Request #44830 · huggingface/transformers

lashahub · 2026-03-18T14:31:45Z

This PR adds AudioFlamingoNext as a separate model name that inherits directly from MusicFlamingo #43538 and keeps the same architecture and behavior.

Changes:

add audioflamingonext model files
register it in the auto mappings
add basic tests for the new model name

HuggingFaceDocBuilderDev · 2026-04-24T18:53:51Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

ebezzam

Some fixes are also needed for the config!

ebezzam · 2026-04-24T19:01:30Z

+        if isinstance(self.audio_config, dict):
+            self.audio_config["model_type"] = self.audio_config.get("model_type", "audioflamingo3_encoder")
+        elif self.audio_config is None:
+            self.audio_config = {"model_type": "audioflamingo3_encoder"}
+        if self.rope_parameters is None:
+            self.rope_parameters = {
+                "rope_type": "default",
+                "rope_theta": kwargs.get("rope_theta", 1200),
+                "partial_rotary_factor": kwargs.get("partial_rotary_factor", 0.2),
+            }
+        if isinstance(self.audio_config, dict):
+            self.audio_config["model_type"] = self.audio_config.get("model_type", "audioflamingonext_encoder")
+            self.audio_config = CONFIG_MAPPING[self.audio_config["model_type"]](**self.audio_config)
+        elif self.audio_config is None:
+            self.audio_config = CONFIG_MAPPING["audioflamingonext_encoder"]()


two code blocks for self.audio_config are getting genetated

ebezzam · 2026-04-24T19:03:26Z

+        if isinstance(self.audio_config, dict):
+            self.audio_config["model_type"] = self.audio_config.get("model_type", "audioflamingo3_encoder")
+        elif self.audio_config is None:
+            self.audio_config = {"model_type": "audioflamingo3_encoder"}


modular doesn't let you override the other self.audio_config unfortunately. But what stops from using MusicFlamingoConfig as is?

lashahub · 2026-04-25T05:13:10Z

@ebezzam thanks for the reviews!

Reverted the Audio Flamingo 3 / Music Flamingo fixture and test changes.
Added the AF-Next fixture reproducer link in the integration tests: https://gist.github.com/lashahub/5dbee78c5faedd5389e211da85e3066d
Simplified the AF-Next config inheritance and removed the duplicated generated audio_config handling.
Kept the Music Flamingo RoPE default setup before super().__post_init__() because the base config now standardizes rope_parameters=None to the global default before model-specific defaults can be applied.
Reran AF-Next slow modeling tests locally on GPU.

eustlb

Thanks a lot for the great work, @lashahub. Since this model is actually MusicFlamingo, there is no strong reason to merge it as a separate model. I understand that the main issue is the naming, and now that we have clarified the situation, it might have been better to merge this architecture from the start under the AudioFlamingoNext naming, with the MusicFlamingo Hub repo simply being a specific checkpoint of the AudioFlamingoNext model.

Note that adding a new model based on an already merged architecture is not a problem. For example, look at the granite-speech releases: since they are just new checkpoints for the granite_speech model type, they simply use AutoModelForSpeechSeq2Seq.

Here, I see two possible paths forward:

The simplest option: just use AutoModelForSpeechSeq2Seq with model_type: musicflamingo in the config.
If the MusicFlamingo checkpoints on the Hub actually correspond to the AudioFlamingoNext architecture, then we could replace the current musicflamingo architecture with AudioFlamingoNext and add MusicFlamingo classes for backward compatibility.

Personally, I would much rather go with option 1.

# Conflicts: # src/transformers/models/musicflamingo/configuration_musicflamingo.py # src/transformers/models/musicflamingo/modular_musicflamingo.py

lashahub · 2026-05-13T02:58:00Z

Thanks for the suggestion @eustlb, I followed option 1.

Audio Flamingo Next is no longer added as a separate Transformers architecture. I removed the AF-Next model/config/processor/tests/docs and switched the AF-Next conversion path to use the existing MusicFlamingo classes with model_type: musicflamingo.

I kept the AF-Next converter under MusicFlamingo for reproducibility, and reconverted the AF-Next checkpoints with this setup.

ebezzam · 2026-05-13T17:41:08Z

        tokenizer=AutoTokenizer.from_pretrained(str(llm_dir), chat_template=tokenizer_chat_template, use_fast=True),
        chat_template=processor_chat_template,
    )
+    processor.tokenizer.model_max_length = 32768


FYI checkpoint is actually using this: https://huggingface.co/nvidia/audio-flamingo-3-hf/blob/main/config.json#L66

ebezzam

@eustlb they've gone forward with the first option 🤗

The model type is updated in the checkpoint: https://huggingface.co/nvidia/audio-flamingo-next-hf/commit/1574fdfdb77dee40f7373e6e0faa16f70c224743

I left some comments on the other changes if we can get your approval?

ebezzam · 2026-05-13T17:44:19Z

@eustlb ok to add the conversion script for the new checkpoint? https://huggingface.co/nvidia/audio-flamingo-next-hf

ebezzam · 2026-05-13T17:45:11Z

        tokenizer=AutoTokenizer.from_pretrained(str(llm_dir), chat_template=tokenizer_chat_template, use_fast=True),
        chat_template=processor_chat_template,
    )
+    processor.tokenizer.model_max_length = 32768


similarly, now matching checkpoint: https://huggingface.co/nvidia/music-flamingo-2601-hf/blob/main/config.json#L76

ebezzam · 2026-05-13T17:54:03Z

@lashahub, @Sreyan88 one other comment: can you replace the path/to/audio.wav in your model card with actual audio paths?

lashahub · 2026-05-13T18:44:10Z

Thanks for the review @ebezzam, I've updated the path placeholders.

eustlb · 2026-05-14T09:01:07Z

+            self.rope_parameters = {
+                "rope_type": "default",
+                "rope_theta": kwargs.get("rope_theta", 1200),
+                "partial_rotary_factor": kwargs.get("partial_rotary_factor", 0.2),
+            }


rope_theta and partial_rotary_factor are not exposed kwargs, API is to directly pass rope_parameters

Suggested change

self.rope_parameters = {

"rope_type": "default",

"rope_theta": kwargs.get("rope_theta", 1200),

"partial_rotary_factor": kwargs.get("partial_rotary_factor", 0.2),

}

self.rope_parameters = {

"rope_type": "default",

"rope_theta": 1200.0,

"partial_rotary_factor": 0.2,

}

github-actions · 2026-05-14T09:03:10Z

[For maintainers] Suggested jobs to run (before merge)

run-slow: audioflamingo3, musicflamingo

* Music flamingo * Fix pos embeddings * Method arg docstrings * Add tests & docs * Fix AF3 dtype bug * Fix the MF performance issue * Fix pos embeddings * Fix embeddings & format * Remove external deps * Update processor token names * Cleanup * Simplify RotaryEmbedding to lang-only * Reuse AF3 config classes * Trim+rename rotary embedding * Call parent _init_weights first and drop rotary einsum * Precompute rotary cache at init * Use modular processor pattern for MusicFlamingo * Remove audio-only inference example * Refactor Audio Feature Casting Path * Clarify private source repo * Clean up modular * Move config to modular * Formatting * Remove dummy * Derive musicflamingo timing and rotary config * Llama style rotary embeddings * Added reproducer comments * Expose _init_weights for modular. * Satisfy repo checks * Align MusicFlamingo rotary with Llama style * Move MusicFlamingo _init_weights to encoder * Keep old behavior * Move MusicFlamingo rotary settings into encoder rope_parameters * Use AutoConfig in AF3/MF * Align MusicFlamingo RoTE with Llama RoPE conventions * Update outdated fixtures * init_weights without changing others * FIx import * Remove backward compat * Regenerate modeling for MF * Fix AF3 batch inference bug * Simplify config and nit. * Conform more to transformers convention, e.g. removing unused code paths. * Add another possible AF3 prefix. * Use auto_docstring and update docstrings. * Nits * Nit for review * Shift RoTE to main model so that encoder can be directly used from AF3. * Refactoring nit. * Fix init * Fix some failing tests * Fix AF3 & MF and add batching tests * Fix audio embedding masking (bad post length) * Nits and remove since same as GLM was bug in post length computation * Simplify MF as AF3, and style checks. * New config after merge and modular update. * Address music flamingo tests, and some cleanup. * style check * Regenerate config. * Update fixtures. * Nits * Nit * Add AudioFlamingoNext alias model * Regenerate AF-Next files * Improve RoTE config * Refine MusicFlamingo rotary time handling * Simplification, and update AF3 processor for better modular * Fix torch export * Simplify modular, including upstreaming input_ids input to get_audio_features * Remove upstreaming of input_ids to get_audio_features, and remove audio_rotary_dim. * Switch to MoonshineRotaryEmbedding, and cleanup. * Remove hardcoded MusicFlamingo partial_rotary_factor * Update fixtures * Compile re.sub * Update src/transformers/models/musicflamingo/modular_musicflamingo.py Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com> * Update src/transformers/models/musicflamingo/modular_musicflamingo.py Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com> * Style * Update fixtures. * Conditional torch import for processor. * Merge MF & convert modular * 30 mins in AF-Next * Regen * Fix AF-Next converter * AF-Next readme * Add tests * Update AF-Next docs * Fix MusicFlamingo and AFNext context metadata and rope defaults * Update fixtures * Address reviews * Add reproducer * Update AF-Next converter * Remove AF-Next * Keep AF-Next converter * Update src/transformers/models/musicflamingo/modular_musicflamingo.py * make --------- Co-authored-by: Eric B <ebezzam@gmail.com> Co-authored-by: Eric Bezzam <4757445+ebezzam@users.noreply.github.com> Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com> Co-authored-by: eustlb <94853470+eustlb@users.noreply.github.com>

lashahub added 30 commits December 24, 2025 10:59

Music flamingo

caf33f1

Fix pos embeddings

b69a9d1

Merge branch 'huggingface:main' into main

b4acc17

Method arg docstrings

cf1e9bc

Add tests & docs

44e801b

Merge branch 'huggingface:main' into main

f0956e3

Fix AF3 dtype bug

c973445

Fix the MF performance issue

e3a17fb

Fix pos embeddings

627dee8

Merge branch 'main' of https://github.com/lashahub/transformers

e9df30d

Fix embeddings & format

4c48132

Remove external deps

d67c114

Update processor token names

aedd341

Cleanup

87d55a9

Simplify RotaryEmbedding to lang-only

be22746

Reuse AF3 config classes

e5b4677

Trim+rename rotary embedding

d7e0bcb

Call parent _init_weights first and drop rotary einsum

74af4fa

Precompute rotary cache at init

d622368

Use modular processor pattern for MusicFlamingo

cab3937

Remove audio-only inference example

9dd94a0

Refactor Audio Feature Casting Path

767a1d5

Clarify private source repo

9119660

Clean up modular

fc8ab3a

Move config to modular

5159745

Formatting

4aae4ff

Remove dummy

ffef7db

Derive musicflamingo timing and rotary config

90148ed

Llama style rotary embeddings

87cb03f

Added reproducer comments

5abac95

ebezzam reviewed Apr 24, 2026

View reviewed changes

lashahub added 2 commits April 25, 2026 00:49

Address reviews

d79fb29

Add reproducer

205f8c8

This was referenced Apr 29, 2026

Cumulative feature and defect updates from recent Transformers PRs evalstate/transformers#42

Open

Cumulative defect fixes from recent Transformers PRs evalstate/transformers#43

Open

eustlb reviewed May 12, 2026

View reviewed changes

lashahub added 4 commits May 12, 2026 14:42

Update AF-Next converter

7e63ddd

Remove AF-Next

e38b6ac

Keep AF-Next converter

8771a9f

Merge branch 'main' into add_AudioFlamingoNext

7091a2a

# Conflicts: # src/transformers/models/musicflamingo/configuration_musicflamingo.py # src/transformers/models/musicflamingo/modular_musicflamingo.py

lashahub changed the title ~~Add AudioFlamingoNext model~~ Support Audio Flamingo Next checkpoints May 13, 2026

Merge branch 'main' into add_AudioFlamingoNext

da61fd3

ebezzam reviewed May 13, 2026

View reviewed changes

ebezzam approved these changes May 13, 2026

View reviewed changes

eustlb approved these changes May 14, 2026

View reviewed changes

eustlb added 2 commits May 14, 2026 11:02

Update src/transformers/models/musicflamingo/modular_musicflamingo.py

88e9644

make

3f1219d

eustlb enabled auto-merge May 14, 2026 09:02

eustlb added this pull request to the merge queue May 14, 2026

Merged via the queue into huggingface:main with commit dca3347 May 14, 2026
22 checks passed

Uh oh!

Conversation

lashahub commented Mar 18, 2026

Uh oh!

HuggingFaceDocBuilderDev commented Apr 24, 2026

Uh oh!

ebezzam left a comment

Choose a reason for hiding this comment

Uh oh!

ebezzam Apr 24, 2026

Choose a reason for hiding this comment

Uh oh!

ebezzam Apr 24, 2026

Choose a reason for hiding this comment

Uh oh!

lashahub commented Apr 25, 2026

Uh oh!

eustlb left a comment

Choose a reason for hiding this comment

Uh oh!

lashahub commented May 13, 2026

Uh oh!

ebezzam May 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ebezzam left a comment

Choose a reason for hiding this comment

Uh oh!

ebezzam May 13, 2026

Choose a reason for hiding this comment

Uh oh!

ebezzam May 13, 2026

Choose a reason for hiding this comment

Uh oh!

ebezzam commented May 13, 2026

Uh oh!

lashahub commented May 13, 2026

Uh oh!

eustlb May 14, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented May 14, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ebezzam May 13, 2026 •

edited

Loading