fix: adapt Whisper models to use self.loss_function with num_items_in_batch support

[Ashutosh-177]

Summary

• Fixes gradient accumulation for Whisper by replacing the manual CrossEntropyLoss instantiation with self.loss_function, which supports the num_items_in_batch parameter introduced in Fix Gradient Accumulation issue #34191.
• Adds an explicit num_items_in_batch parameter to both WhisperForConditionalGeneration.forward() and WhisperForCausalLM.forward().
• Sets self.loss_type = "ForMaskedLM" in both Whisper class __init__ methods so the correct (no-shift) loss is used without touching the global LOSS_MAPPING.

Changes

src/transformers/models/whisper/modeling_whisper.py

• WhisperForConditionalGeneration.__init__: sets self.loss_type = "ForMaskedLM". Encoder-decoder models receive labels already aligned with decoder logits — no token shifting is needed.
• WhisperForConditionalGeneration.forward(): adds num_items_in_batch parameter; replaces manual CrossEntropyLoss block with self.loss_function(...).
• WhisperForCausalLM.__init__: same self.loss_type = "ForMaskedLM" override, preserving the original no-shift behavior.
• WhisperForCausalLM.forward(): same treatment.

src/transformers/loss/loss_utils.py

• No global mapping changes — LOSS_MAPPING["ForConditionalGeneration"] remains ForCausalLMLoss to avoid affecting decoder-only models like LlavaForConditionalGeneration and PaliGemmaForConditionalGeneration that rely on label shifting.

Behavior

• When num_items_in_batch=None (default): loss is identical to the previous CrossEntropyLoss(reduction="mean") call — no behavior change.
• When num_items_in_batch is provided by the Trainer: loss is normalized correctly across gradient accumulation steps, fixing the GA bug.

Test plan

• Verify loss is unchanged when num_items_in_batch=None
• Verify loss is correctly normalized when num_items_in_batch is supplied
• Run existing Whisper tests: pytest tests/models/whisper/ -x

Closes #36119

[Copilot]

Pull request overview

This PR updates Whisper’s forward() implementations to use the unified self.loss_function(...) interface (including num_items_in_batch support) to fix gradient-accumulation loss scaling, and adjusts the global loss dispatch for ForConditionalGeneration.

Changes:

• Add num_items_in_batch to WhisperForConditionalGeneration.forward() and WhisperForCausalLM.forward() and route loss computation through self.loss_function(...).
• Update LOSS_MAPPING["ForConditionalGeneration"] to point to ForMaskedLMLoss instead of ForCausalLMLoss.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
src/transformers/models/whisper/modeling_whisper.py	Switch Whisper LM losses to self.loss_function(...) and accept num_items_in_batch for correct GA normalization.
src/transformers/loss/loss_utils.py	Changes the default loss implementation used for classes matching ForConditionalGeneration.

Comments suppressed due to low confidence (1)

src/transformers/models/whisper/modeling_whisper.py:1224

• WhisperForCausalLM previously computed loss directly against labels (no shifting). Switching to self.loss_function will use ForCausalLMLoss by default, which shifts labels internally; this is a behavior change and contradicts the PR description claim of “no behavior change when num_items_in_batch is None”. If Whisper’s decoder labels are intended to be already aligned with logits (as in many encoder-decoder decoders), consider setting self.loss_type = "ForMaskedLM" for this class or passing an explicit shift_labels to preserve current semantics.

        logits = self.proj_out(outputs[0])

        loss = None
        if labels is not None:
            loss = self.loss_function(
                logits=logits, labels=labels, vocab_size=self.config.vocab_size, num_items_in_batch=num_items_in_batch
            )

[github-actions]

[For maintainers] Suggested jobs to run (before merge)

run-slow: whisper

[Ashutosh-177]

Opened PR #46068 that fixes this. All CI checks are passing — would appreciate a review.