Add torchcodec in docstrings/tests for `datasets` 4.0 by lhoestq · Pull Request #39156 · huggingface/transformers

lhoestq · 2025-07-01T16:30:53Z

continuation of #39060 (so this PR also contains the object detection fix so we can run the CI with datasets on main)

it also adds support for torchcodec.decoders.AudioDecoder as input to audio classification and asr pipelines

HuggingFaceDocBuilderDev · 2025-07-01T16:48:57Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

ydshieh · 2025-07-01T20:44:05Z

triggered https://github.com/huggingface/transformers/actions/runs/16009282798

ydshieh · 2025-07-02T08:50:13Z

Hi @lhoestq There is slack reporting issues, but I saw many failed tests

FAILED tests/models/clvp/test_modeling_clvp.py::ClvpModelForConditionalGenerationTest::test_gradient_checkpointing_backward_compatibility - ImportError: To support decoding audio data, please install 'torchcodec'.

I only add

      - name: Install datasets main
        working-directory: /transformers
        run: python3 -m pip install --no-cache-dir git+https://github.com/huggingface/datasets.git@main

maybe I should also install something else?

Could you provide a full installation steps 🙏 ? I will rerun the CI again today.

It's best to verify it before merge, as we have several events like switching all jobs to A10, run with torch 2.8 etc. and they already cause quite some chaos.

lhoestq · 2025-07-02T12:31:08Z

added the missing torchcodec installations, it should be fine now !

ydshieh · 2025-07-03T09:34:27Z

the examples_torch job has

        if isinstance(data_args.train_val_split, float) and data_args.train_val_split > 0.0:
            split = dataset["train"].train_test_split(data_args.train_val_split, seed=training_args.seed)
            dataset["train"] = split["train"]
            dataset["validation"] = split["test"]
    
        # Get dataset categories and prepare mappings for label_name <-> label_id
>       categories = dataset["train"].features["objects"]["category"].feature.names
E       TypeError: 'Sequence' object is not subscriptable

so maybe from datasets ??

Also a tiny change to fix

https://app.circleci.com/pipelines/github/huggingface/transformers/136595/workflows/5580c370-0b69-49f5-a3e9-7c96485eab15/jobs/1810677

is also necessary. (update the 1 src/transformers/models/hubert/modeling_hubert.py or src/transformers/models/hubert/modular_hubert.py)

lhoestq · 2025-07-03T09:42:43Z

I fixed the TypeError and the modeling_hubert :)

lhoestq · 2025-07-03T09:50:45Z

-...     speech, _ = sf.read(batch["file"])
-...     batch["speech"] = speech
+...     decoder = AudioDecoder(batch["file"])
+...     batch["speech"] = torch.mean(decoder.get_all_samples().data, axis=0)


cc @eustlb since datasets is now using torchcodec I updated this part of the docs (no need to have both soundfile and torchcodec), but since torchcodec isn't available for windows users I might revert, wdyt ?

(shouldn't necessarily block this PR since the goal is to make the CI happy for datasets release, happy to revert later)

I agree that it's better not updating the default docs until it is supported also on windows. I just realised those map_to_array everywhere are broken anyway...

I reverted them (no torchcodec) and then I fixed them since the code docs example on main was actually broken

qubvel

Thanks, LGTM for object detection changes

qubvel · 2025-07-03T14:20:09Z

 ENV UV_PYTHON=/usr/local/bin/python
 RUN pip --no-cache-dir install uv && uv venv && uv pip install --no-cache-dir -U pip setuptools
-RUN uv pip install --no-cache-dir 'torch' 'torchaudio' 'torchvision' --index-url https://download.pytorch.org/whl/cpu
+RUN uv pip install --no-cache-dir 'torch' 'torchaudio' 'torchvision' 'torchcodec' --index-url https://download.pytorch.org/whl/cpu


should we add it to any requirements instead?

It's also in the requirements, at examples/pytorch/_tests_requirements.txt

I assume having it in the dockerfile with the other torch-related libs is about making sure it's the CPU version that is installed

eustlb

Thanks @lhoestq! I love torchcodec! (it would be amazing in the near future to move from numpy arrays by default → torch to everything torch native?). Comments:

can we revert doc changes? snippets for the few I tried are broken anyway, let's change to torchcodec when it is not blocking for windows users
why the stereo → mono conversion in the pipeline? AFAIK it is directly handled by the models no?

eustlb · 2025-07-03T16:52:13Z

-...     speech, _ = sf.read(batch["file"])
-...     batch["speech"] = speech
+...     decoder = AudioDecoder(batch["file"])
+...     batch["speech"] = torch.mean(decoder.get_all_samples().data, axis=0)


I agree that it's better not updating the default docs until it is supported also on windows. I just realised those map_to_array everywhere are broken anyway...

eustlb · 2025-07-03T17:08:34Z

+                _data = torch.mean(_data, 0) if _data.ndim > 1 else _data
+                inputs = {"array": _data, "sampling_rate": _audio_samples.sample_rate}


AFAIK we do not do mono → stereo conversion here by default has it's directly handled by the model's feature extractor (eg here). Have you added it here for any reason?

done ! ok to merge ?

lhoestq · 2025-07-04T10:02:58Z

can we revert doc changes? snippets for the few I tried are broken anyway, let's change to torchcodec when it is not blocking for windows users

sure !

why the stereo → mono conversion in the pipeline? AFAIK it is directly handled by the models no?

oh I didn't know that, let me change it :)

I was converting to mono because the {"array": ..., "sampling_rate": ...} input format was already mono so I assumed the models were also mono.

eustlb

Tiny nit, otherwise LGTM! 🤗

eustlb · 2025-07-08T10:13:16Z

@@ -436,9 +436,10 @@ def forward(
        input_values (`torch.FloatTensor` of shape `(batch_size, max_length, num_mel_bins)`):
            Float values mel features extracted from the raw audio waveform. Raw audio waveform can be obtained by
            loading a `.flac` or `.wav` audio file into an array of type `list[float]` or a `numpy.ndarray`, *e.g.* via


Can we propagate this change?

Suggested change

loading a `.flac` or `.wav` audio file into an array of type `list[float]` or a `numpy.ndarray`, *e.g.* via

loading a `.flac` or `.wav` audio file into an array of type `list[float]`, a `numpy.ndarray` or a `torch.Tensor`, *e.g.* via

github-actions · 2025-07-08T14:00:55Z

[For maintainers] Suggested jobs to run (before merge)

run-slow: audio_spectrogram_transformer, clvp, data2vec, hubert, moonshine, qwen2_5_omni, qwen2_audio, sew, sew_d, speech_encoder_decoder, speech_to_text

) * fix dataset run_object_detection * bump version * keep same dataset actually * torchcodec in docstrings and testing utils * torchcodec in dockerfiles and requirements * remove duplicate * add torchocodec to all the remaining docker files * fix tests * support torchcodec in audio classification and ASR * [commit to revert] build ci-dev images * [commit to revert] trigger circleci * [commit to revert] build ci-dev images * fix * fix modeling_hubert * backward compatible run_object_detection * revert ci trigger commits * fix mono conversion and support torch tensor as input * revert map_to_array docs + fix it * revert mono * nit in docstring * style * fix modular --------- Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>

lhoestq added 5 commits June 26, 2025 16:56

fix dataset run_object_detection

b5c2c0e

bump version

f4cdecd

keep same dataset actually

fb4952d

torchcodec in docstrings and testing utils

e207f35

torchcodec in dockerfiles and requirements

35f906e

lhoestq mentioned this pull request Jul 1, 2025

Make run_object_detection compatible with datasets 4.0 #39060

Closed

remove duplicate

d56f55f

add torchocodec to all the remaining docker files

b05f248

lhoestq and others added 6 commits July 2, 2025 22:31

fix tests

4339bd7

support torchcodec in audio classification and ASR

1bcaf5f

[commit to revert] build ci-dev images

4b4b445

[commit to revert] trigger circleci

6c4b937

[commit to revert] build ci-dev images

aa132d0

fix

49e4f92

lhoestq added 2 commits July 3, 2025 11:40

fix modeling_hubert

7e36d30

backward compatible run_object_detection

3514289

lhoestq commented Jul 3, 2025

View reviewed changes

revert ci trigger commits

ad030a8

lhoestq marked this pull request as ready for review July 3, 2025 12:27

fix mono conversion and support torch tensor as input

c9d463e

qubvel approved these changes Jul 3, 2025

View reviewed changes

eustlb reviewed Jul 3, 2025

View reviewed changes

revert map_to_array docs + fix it

69bcb8e

revert mono

0649763

eustlb approved these changes Jul 8, 2025

View reviewed changes

lhoestq added 3 commits July 8, 2025 14:36

nit in docstring

b1627f9

style

090695e

fix modular

2b364c3

lhoestq merged commit 1ecd52e into main Jul 8, 2025
26 checks passed

lhoestq deleted the fix-dataset-run_object_detection-and-add-torchcodec branch July 8, 2025 15:06

lhoestq mentioned this pull request Jul 9, 2025

Fix audio pipeline with torchcodec input #39309

Open

		_data = torch.mean(_data, 0) if _data.ndim > 1 else _data
		inputs = {"array": _data, "sampling_rate": _audio_samples.sample_rate}

	loading a `.flac` or `.wav` audio file into an array of type `list[float]` or a `numpy.ndarray`, e.g. via
	loading a `.flac` or `.wav` audio file into an array of type `list[float]`, a `numpy.ndarray` or a `torch.Tensor`, e.g. via

Uh oh!

Conversation

lhoestq commented Jul 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HuggingFaceDocBuilderDev commented Jul 1, 2025

Uh oh!

ydshieh commented Jul 1, 2025

Uh oh!

ydshieh commented Jul 2, 2025

Uh oh!

lhoestq commented Jul 2, 2025

Uh oh!

ydshieh commented Jul 3, 2025

Uh oh!

lhoestq commented Jul 3, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

qubvel left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eustlb left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lhoestq commented Jul 4, 2025

Uh oh!

eustlb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jul 8, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

lhoestq commented Jul 1, 2025 •

edited

Loading

eustlb left a comment •

edited

Loading