[Whisper] Fix forced decoder ids

[sanchit-gandhi]

What does this PR do?

The Whisper tokenizer has a property self.prefix_tokens that returns the token ids appended to the start of label sequence:

<|startoftranscript|> <|lang_id|> <|task|> <|notimestamps|> ...

In the PR #20589, the method get_decoder_prompt_ids was copied from the Whisper processor to the Whisper tokenizer, where it then made use of the tokenizer property self.prefix_tokens. The method get_decoder_prompt_ids is used to set the tokens that are forced at the beginning of the generation process.

However, the forced decoder ids should not contain the <|startoftranscript|> token: this is the decoder_start_token_id that we use as token 0 when we start generation. If we include <|startoftranscript|> in our forced decoder ids, we'll get a double generation of <|startoftranscript|>. Thus, we only want to set the following tokens in the forced_decoder_ids:

<|lang_id|> <|task|> <|notimestamps|> ...

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[ArthurZucker]

LGTM, we are just gonna get whispered at again by @ydshieh for failing tests 😩 🤣

[ArthurZucker]

Haha nice the test was indeed worse it

[sanchit-gandhi]

fyi @sgugger, the final fix we hope 🤞

[ydshieh]

LGTM, but could you explain this a bit:

the start of label sequence:
<|startoftranscript|> <|lang_id|> <|task|> <|notimestamps|> ...

My general understanding is that, the lable sequence should NOT contain the decoder_start_token_id (here is <|startoftranscript|>).

But here you mention the start of label sequence -- I have some doubt here.

[sanchit-gandhi]

Yes! Let me clarify!

When training, we need to encode a sentence to a sequence of label ids. Here, we need to append the 'special' beginning of sentence tokens to the label ids. This is so that the model learns to predict the correct 'special' tokens for the generation process. For a full list of the tokens added, see this PR: #19921

One of these tokens is the <|startoftranscript|> token. This is consistent with other tokenisers in the library, such as the BART tokeniser:

from transformers import BartTokenizer

tokenizer = BartTokenizer.from_pretrained("facebook/bart-base")
input_str = "the cat"
tokens = tokenizer(input_str).input_ids
print(tokenizer.decode(tokens))

Print Output:

<s>the cat</s>

Now, it doesn't matter for training whether or not we append the decoder start token id to the start of our label sequence, because we cut it in our data collator:

transformers/examples/pytorch/speech-recognition/run_speech_recognition_seq2seq.py

Line 249 in 3ac040b

# if bos token is appended in previous tokenization step,

So, adding the decoder start token id is more for making the tokeniser user friendly and consistent with other tokenisers in the library.

[sgugger]

Thanks for fixing!

[ydshieh]

@sanchit-gandhi Thanks. Just want to point out: For bart, yes, we have bos <s> (id 0). But it is not the decoder start token (which is </s> for bart, with id 2) - it is just the start of the sentence (not ready for generation). The labels has bos but not decoder_start_token. The labels will be shifted and prepended with </s> to become decoder input ids.

In Whisper, I understand we want to be user-friendly. And as you have cut it in data collator, it is fine. But IMO, this is something a bit different from our NLP models (i.e. Bart here). Hopefully I understand it correctly.