Implement VibeVoice by pengzhiliang · Pull Request #40546 · huggingface/transformers

pengzhiliang · 2025-08-29T12:47:34Z

What does this PR do?

Merge the model from https://github.com/microsoft/VibeVoice/tree/main

HF:
https://huggingface.co/microsoft/VibeVoice-1.5B

update

ebezzam

@pengzhiliang thanks for the PR! This is an exciting model to add 🔥

My first comments are mainly on rearranging content to be consistent with the other models in Transformers, and creating a modular file to better optimize copying components from other models in Transformers.

There are also some other files to modify:

in src/transformers/models/auto
in docs
and eventually some tests (for which a lot of code can be copied from other models)

As an example of typical files to create/modify, you can check out the Qwen2.5-Omni PR, which is also multimodal.

fakerybakery · 2025-09-06T22:37:22Z

Hopefully this PR can get merged, since the deletion of the VibeVoice repo not sure the original authors can continue contributing to this PR

fakerybakery · 2025-10-15T01:33:05Z

Glad to see that someone has picked this PR up, thanks 🤗

github-actions · 2026-06-30T16:20:52Z

CI recap

Dashboard: View test results in Grafana
Latest run: 28458306538:2
Result: failure | Jobs: 13 | Tests: 44,325 | Failures: 57 | Duration: 14h 22m

ebezzam · 2026-06-30T16:36:41Z

run-slow: vibevoice

github-actions · 2026-06-30T16:38:05Z

Workflow Run ⚙️

This comment contains run-slow, running the specified jobs:

models: ["models/vibevoice"]
quantizations: []

github-actions · 2026-06-30T16:53:00Z

CI Results

Workflow Run ⚙️

Commit Info

Context	Commit	Description
RUN	36208e7a	workflow commit (merge commit)
PR	bbe259f6	branch commit (from PR)
main	181beb3b	base commit (on `main`)

✅ No failing test specific to this PR 🎉 👏 !

vasqu

Some initial comments from my side 🤗 mainly took a look at generation/modeling for now

vasqu · 2026-07-02T15:58:36Z

+- [bezzam/VibeVoice-1.5B-hf](https://huggingface.co/bezzam/VibeVoice-1.5B-hf)
+- [bezzam/VibeVoice-7B-hf](https://huggingface.co/bezzam/VibeVoice-7B-hf)


Rebump before it gets lost, ig a community org or we have contact to transfer?

vasqu · 2026-07-02T16:07:41Z

+noise_scheduler = diffusers.DPMSolverMultistepScheduler(
+    beta_schedule="squaredcos_cap_v2",
+    prediction_type="v_prediction",
+)
+model.generation_config.noise_scheduler = noise_scheduler
+audio = model.generate(**inputs)


My main idea would be to add a custom dict like entry that we resolve for vibevoice only as custom entries are passed along and saved so we can make use of that and integrate it into our own generate pipeline in the end --> create the scheduler at runtime then

vasqu · 2026-07-02T17:06:06Z

+            Padding mask to remove padded parts of audio.
+        """
+
+        with torch.no_grad():


taken from original and because the acoustic tokenizer (audio tower) is only meant for inference within this architecture, even during training ("During VIBEVOICE training, the pre-trained acoustic and semantic tokenizers remained frozen, with only the LLM and diffusion head parameters being learnable", bottom of p.3 of paper).

is there better way to do this?

Let's add a small comment but should be fine. Was just surprised

vasqu · 2026-07-02T17:06:26Z

+        inputs_embeds: torch.FloatTensor | None = None,
+        input_values: torch.FloatTensor | None = None,
+        padding_mask: torch.BoolTensor | None = None,
+        **kwargs,


typing of kwargs

vasqu · 2026-07-02T17:07:22Z

+        num_diffusion_steps (`int`, *optional*, defaults to config.num_diffusion_steps):
+            For training, the number of diffusion steps to use.


hmm ok here it is, let's avoid the double trouble and only have it as config value?

how about only as a forward parameter? like ddpm_batch_multiplier and noise_scheduler so all training-related parameters are passed this way

Yep glad with either version just dont want both 😬

makes sense!

Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com>

github-actions · 2026-07-03T08:45:28Z

Thank you for your contribution 🤗!

CI Security Gate — automatic approval blocked

This PR was not automatically approved for CI because the security gate failed.

Possible reasons:

The PR touches 50 or more files — only PRs with fewer than 50 changed files are automatically approved
A changed file is outside the allowed directories (src/, tests/, docs/, utils/), has a disallowed extension (only .py, .txt, .md permitted outside tests/ and docs/), or is not .md/.yml inside docs/
A new high-severity security issue was detected in the changed Python files (Bandit check)

See the workflow run for the exact violations.

A maintainer can review and manually approve CI if a finding is a false positive.

github-actions · 2026-07-03T08:46:08Z

[For maintainers] Suggested jobs to run (before merge)

run-slow: auto, csm, vibevoice, vibevoice_acoustic_tokenizer

Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com>

github-actions · 2026-07-03T09:01:00Z

Thank you for your contribution 🤗!

CI Security Gate — automatic approval blocked

This PR was not automatically approved for CI because the security gate failed.

Possible reasons:

The PR touches 50 or more files — only PRs with fewer than 50 changed files are automatically approved
A changed file is outside the allowed directories (src/, tests/, docs/, utils/), has a disallowed extension (only .py, .txt, .md permitted outside tests/ and docs/), or is not .md/.yml inside docs/
A new high-severity security issue was detected in the changed Python files (Bandit check)

See the workflow run for the exact violations.

A maintainer can review and manually approve CI if a finding is a false positive.

github-actions · 2026-07-03T09:01:49Z

[For maintainers] Suggested jobs to run (before merge)

run-slow: auto, csm, vibevoice, vibevoice_acoustic_tokenizer

github-actions · 2026-07-03T09:02:25Z

Thank you for your contribution 🤗!

CI Security Gate — automatic approval blocked

This PR was not automatically approved for CI because the security gate failed.

Possible reasons:

The PR touches 50 or more files — only PRs with fewer than 50 changed files are automatically approved
A changed file is outside the allowed directories (src/, tests/, docs/, utils/), has a disallowed extension (only .py, .txt, .md permitted outside tests/ and docs/), or is not .md/.yml inside docs/
A new high-severity security issue was detected in the changed Python files (Bandit check)

See the workflow run for the exact violations.

A maintainer can review and manually approve CI if a finding is a false positive.

github-actions · 2026-07-03T09:03:15Z

[For maintainers] Suggested jobs to run (before merge)

run-slow: auto, csm, vibevoice, vibevoice_acoustic_tokenizer

github-actions · 2026-07-03T09:07:37Z

Thank you for your contribution 🤗!

CI Security Gate — automatic approval blocked

This PR was not automatically approved for CI because the security gate failed.

Possible reasons:

The PR touches 50 or more files — only PRs with fewer than 50 changed files are automatically approved
A changed file is outside the allowed directories (src/, tests/, docs/, utils/), has a disallowed extension (only .py, .txt, .md permitted outside tests/ and docs/), or is not .md/.yml inside docs/
A new high-severity security issue was detected in the changed Python files (Bandit check)

See the workflow run for the exact violations.

A maintainer can review and manually approve CI if a finding is a false positive.

github-actions · 2026-07-03T09:08:23Z

[For maintainers] Suggested jobs to run (before merge)

run-slow: auto, csm, vibevoice, vibevoice_acoustic_tokenizer

Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com>

github-actions · 2026-07-03T09:52:22Z

Thank you for your contribution 🤗!

CI Security Gate — automatic approval blocked

This PR was not automatically approved for CI because the security gate failed.

Possible reasons:

The PR touches 50 or more files — only PRs with fewer than 50 changed files are automatically approved
A changed file is outside the allowed directories (src/, tests/, docs/, utils/), has a disallowed extension (only .py, .txt, .md permitted outside tests/ and docs/), or is not .md/.yml inside docs/
A new high-severity security issue was detected in the changed Python files (Bandit check)

See the workflow run for the exact violations.

A maintainer can review and manually approve CI if a finding is a false positive.

github-actions · 2026-07-03T09:53:11Z

[For maintainers] Suggested jobs to run (before merge)

run-slow: auto, csm, vibevoice, vibevoice_acoustic_tokenizer

github-actions · 2026-07-03T16:37:20Z

Thank you for your contribution 🤗!

CI Security Gate — automatic approval blocked

This PR was not automatically approved for CI because the security gate failed.

Possible reasons:

The PR touches 50 or more files — only PRs with fewer than 50 changed files are automatically approved
A changed file is outside the allowed directories (src/, tests/, docs/, utils/), has a disallowed extension (only .py, .txt, .md permitted outside tests/ and docs/), or is not .md/.yml inside docs/
A new high-severity security issue was detected in the changed Python files (Bandit check)

See the workflow run for the exact violations.

A maintainer can review and manually approve CI if a finding is a false positive.

vasqu

Some more comments in the meantime, did not go full deep dive but still a few details here and there

vasqu · 2026-07-08T19:35:42Z

+print(f"Saved output to {fn}")
+```
+
+### Batched inference


Yup much better imo!

vasqu · 2026-07-08T19:36:58Z

+# Compute losses
+lm_loss = outputs.loss
+diffusion_loss = outputs.diffusion_loss
+total_loss = lm_loss + diffusion_loss


this is weird why are we outputting both separately and not directly the final one? Otherwise the trainer will also not work OOB

I think it's better to output separately, because some training schemes may weight them differently as training progresses.

I'll put a comment that this is a sum just for demo purposes

vasqu · 2026-07-08T19:38:35Z

+        _ = model(**inputs)
+torch.cuda.synchronize()
+print(f"Warmup complete in {time.time() - warmup_start:.2f}s.")
+# Warmup complete in 46.39s


feels like we should add a final forward after the warmup?

vasqu · 2026-07-08T19:40:38Z

+        device = kwargs.get("device")
+        processors.append(VibeVoiceTokenConstraintProcessor(valid_tokens, device=device))


Hmm self.device maybe? Or is this the better way, havent seen this in kwargs tbh

vasqu · 2026-07-08T19:42:03Z

+        if noise_scheduler is None:
+            raise ValueError(
+                "VibeVoice generation requires a `noise_scheduler` to be provided, e.g., "
+                "`diffusers.DPMSolverMultistepScheduler(beta_schedule='squaredcos_cap_v2', prediction_type='v_prediction')`."
+            )
+        if not (
+            hasattr(noise_scheduler, "set_timesteps")
+            and hasattr(noise_scheduler, "step")
+            and hasattr(noise_scheduler, "timesteps")
+        ):
+            raise ValueError(
+                f"The provided noise_scheduler ({type(noise_scheduler).__name__}) is not compatible with VibeVoice "
+                "generation. It must implement `set_timesteps` and `step` methods, and have a `timesteps` attribute."
+            )


Yup, it looked very awkward in the docs tbh

vasqu · 2026-07-08T19:55:16Z

+    base_model_tp_plan = {
+        "language_model.layers.*.self_attn.q_proj": "colwise",
+        "language_model.layers.*.self_attn.k_proj": "colwise",
+        "language_model.layers.*.self_attn.v_proj": "colwise",
+        "language_model.layers.*.self_attn.o_proj": "rowwise",
+        "language_model.layers.*.mlp.gate_proj": "colwise",
+        "language_model.layers.*.mlp.up_proj": "colwise",
+        "language_model.layers.*.mlp.down_proj": "rowwise",
+    }


Hmm does it not automatically be applied on submodels with the correct config? Can you double check?

yep can do without

vasqu · 2026-07-08T19:55:45Z

+
+    # NOTE (ebezzam) for modular usage of `LlamaMLP`
+    @property
+    def hidden_size(self) -> int:


should we have a setter?

or actually can avoid this altogether by doing

class VibeVoiceMLP(LlamaMLP): def __init__(self, config): super().__init__(config) self.hidden_size = config.text_config.hidden_size

vasqu · 2026-07-08T19:56:05Z

+    num_head_layers: int = 4
+    intermediate_size: int = 4608
+    rms_norm_eps: float = 1e-5
+    hidden_act: str = "silu"
+    frequency_embedding_size: int = 256
+    diffusion_max_period: int = 10000
+    mlp_bias: bool = False


Might be a big nit but maybe also a subconfig?

vasqu · 2026-07-08T19:58:16Z

+
+@auto_docstring
+class VibeVoicePreTrainedModel(VoxtralPreTrainedModel):
+    _supports_cache_class = False


Hmm so we have no cache or more like we have a super custom cache?

pengzhiliang added 4 commits May 23, 2025 19:24

Merge pull request #1 from huggingface/main

db16649

update

Merge branch 'huggingface:main' into main

bc681cd

Merge branch 'huggingface:main' into main

30f09dd

Merge VibeVoice model

6e6e60f

ebezzam added New model Audio labels Aug 29, 2025

ebezzam requested changes Aug 29, 2025

View reviewed changes

twobob mentioned this pull request Sep 5, 2025

[Feature] [New Model]: VibeVoice-1.5B : A Frontier Long Conversational Text-to-Speech Model sgl-project/sglang#9697

Closed

2 tasks

ebezzam added 8 commits September 23, 2025 17:40

Restructuring.

5f461b6

Formatting.

c91497f

Clean up processor and feature extractor.

0bef8e3

Move audio processing to __call__.

f6893e2

Clean up tokenizer, use modular.

f508927

Move token sequence prep to __call__.

67fcd3e

Move batch prep to __call__.

90490d1

More efficient tokenization.

8698bb5

Galigator added a commit to Galigator/transformers that referenced this pull request Oct 8, 2025

Apply PR huggingface#40546

dca2cc5

ebezzam added 10 commits October 10, 2025 18:59

Start separate acoustic tokenizer model.

92d7a7d

Add init for acoustic tokenizer.

add4e35

Remove audio tokenizer from main model.

396bb2c

Separate semantic model.

fa6100c

Switch to modular and remove debug path.

24e9f13

Start conversion script for main model, clean up semantic modeling.

84c27ad

Update semantic modular.

d7d409e

Single conversion script.

06313d0

Clean up semantic tokenizer modeling.

4af74e9

Update modular.

47fdd93

More semantic cleanup.

8de03e7

ebezzam and others added 2 commits June 30, 2026 18:02

Better modular

3b9bc08

Merge branch 'main' into main

bbe259f

vasqu reviewed Jul 2, 2026

View reviewed changes

Apply suggestion from @vasqu

4b6d2c6

Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com>

Apply suggestions from code review

daee226

Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com>

ebezzam added 2 commits July 3, 2026 11:01

Address doc comments.

5e9d535

Merge branch 'main' of github.com:pengzhiliang/transformers into main

4846128

Address conversion script comment.

2e8e600

Apply suggestions from code review

041008b

Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com>

ebezzam added 4 commits July 3, 2026 15:09

Better api for noise scheduler and monitor_progress

001abfd

Address rest of generate comments.

60160cf

Address rest of comments.

2b05f0d

Merge branch 'main' of github.com:pengzhiliang/transformers into main

80711e7

ebezzam requested a review from vasqu July 3, 2026 16:42

vasqu reviewed Jul 8, 2026

View reviewed changes

		- [bezzam/VibeVoice-1.5B-hf](https://huggingface.co/bezzam/VibeVoice-1.5B-hf)
		- [bezzam/VibeVoice-7B-hf](https://huggingface.co/bezzam/VibeVoice-7B-hf)

		num_diffusion_steps (`int`, optional, defaults to config.num_diffusion_steps):
		For training, the number of diffusion steps to use.

		device = kwargs.get("device")
		processors.append(VibeVoiceTokenConstraintProcessor(valid_tokens, device=device))

Uh oh!

Conversation

pengzhiliang commented Aug 29, 2025 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Uh oh!

ebezzam left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fakerybakery commented Sep 6, 2025

Uh oh!

fakerybakery commented Oct 15, 2025

Uh oh!

github-actions Bot commented Jun 30, 2026

CI recap

Uh oh!

ebezzam commented Jun 30, 2026

Uh oh!

github-actions Bot commented Jun 30, 2026

Uh oh!

github-actions Bot commented Jun 30, 2026

CI Results

Commit Info

Uh oh!

vasqu left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jul 3, 2026

CI Security Gate — automatic approval blocked

Uh oh!

github-actions Bot commented Jul 3, 2026

Uh oh!

github-actions Bot commented Jul 3, 2026

CI Security Gate — automatic approval blocked

Uh oh!

github-actions Bot commented Jul 3, 2026

Uh oh!

github-actions Bot commented Jul 3, 2026

CI Security Gate — automatic approval blocked

Uh oh!

github-actions Bot commented Jul 3, 2026

Uh oh!

github-actions Bot commented Jul 3, 2026

CI Security Gate — automatic approval blocked

Uh oh!

pengzhiliang commented Aug 29, 2025 •

edited by github-actions Bot

Loading