[Whisper Tokenizer] Make more user-friendly by sanchit-gandhi · Pull Request #19921 · huggingface/transformers

sanchit-gandhi · 2022-10-27T09:23:41Z

What does this PR do?

Fixes #19864.

In summary, the Whisper tokenizer is modified to prepend several tokens to the start-of-sequence:

BOS token id (<|startoftranscript|>) -> consistent with other sequence-to-sequence models such as BART.
Language token id (e.g. <|es|> for Spanish) -> set only when the tokenizer is instantiated with argument language=X. Otherwise omitted.
Task token id (e.g. <|translate|> for speech translation) -> set only when the tokenizer is instantiated with argument task=Y. Otherwise omitted.
No time stamps id (<|notimestamps|>) ->set only when the tokenizer is instantiated with argument predict_timestamps=False. For predict_timestamps=True, it is omitted.

In addition, it is modified to always append the end-of-sequence token to the end of the label sequence (<|endoftext|>).

The updated tokenizer behaves as follows:

from transformers import WhisperTokenizer
tokenizer = WhisperTokenizer.from_pretrained("openai/whisper-tiny", language="english", task="transcribe", predict_timestamps=False)

input_ids = tokenizer("hey").input_ids

text_with_special = tokenizer.decode(input_ids, skip_special_tokens=False)
text = tokenizer.decode(input_ids, skip_special_tokens=True)

print("Input ids :", input_ids)
print("Text w/ special :", text_with_special)
print("Text :", text)

Print Output:

Input ids : [50258, 50259, 50359, 50363, 17230, 50257]
Text w/ special : <|startoftranscript|><|en|><|transcribe|><|notimestamps|>hey<|endoftext|>
Text : hey

The attention mask functionality of the Whisper tokenizer is retained (c.f. #19864 (comment)).

Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you read the contributor guideline,
Pull Request section?
Was this discussed/approved via a Github issue or the forum? Please add a link
to it if that's the case.
Did you make sure to update the documentation with your changes? Here are the
documentation guidelines, and
here are tips on formatting docstrings.
Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

HuggingFaceDocBuilderDev · 2022-10-27T09:42:13Z

The documentation is not available anymore as the PR was closed or merged.

sanchit-gandhi · 2022-10-27T18:34:26Z

    def test_tokenizer_special(self):
-        multilingual_tokenizer = WhisperTokenizer.from_pretrained("openai/whisper-tiny.en")
-        text = "<|startoftranscript|>Hey! How are you feeling? J'ai l'impression que 郷さん est prêt<|endoftext|>"
+        multilingual_tokenizer = WhisperTokenizer.from_pretrained(


Refactored to use a multilingual tokenizer and changed the expected ID's accordingly.

sanchit-gandhi · 2022-10-27T18:35:37Z

    def test_batch_encoding(self):
-        multilingual_tokenizer = WhisperTokenizer.from_pretrained("openai/whisper-tiny.en")
-        batch = ["<|en|><|notimestamps|>", "<|en|><|notimestamps|>I am sure that"]
+        multilingual_tokenizer = WhisperTokenizer.from_pretrained(


Refactored to use a multilingual tokeniser and changed the expected ID's accordingly.

sanchit-gandhi · 2022-10-27T18:37:39Z

    return pairs


+LANGUAGES = {


Currently using language (e.g. "spanish") instead of lang_id (e.g. "es") -> this is how the original Whisper model does it. If there's a preference for lang_id I'm happy to switch!

ArthurZucker

Awesome work here! A few nits here an there but thanks a lot
Do you think we can also update the doc or write somewhere on the model card that in order to train the model, just pass the language and run it.

ArthurZucker · 2022-10-27T18:40:40Z

+        all_special_ids = self.all_special_ids
+        bos_token_id = all_special_ids[-106]
+        translate_token_id = all_special_ids[-6]
+        transcribe_token_id = all_special_ids[-5]
+        notimestamps_token_id = all_special_ids[-1]


nit : Not really a fan of chard coded indexes. Maybe using the "all_special_tokens" makes it a bit more readable.

I see your point! The only downside of using all_special_tokens is that we'll have to do an extra tokenisation step of converting from tokens -> id's in this method:

bos_token = all_special_tokens[-106] translate_token = all_special_tokens[-6] ... # get prefix tokens (bos_token, lang_token, task_token, notimestamps_token) ... prefix_ids = self.encode(prefix_tokens) # <- extra step to convert from tokens to ids

Co-authored-by: ArthurZucker <arthur@huggingface.co>

sanchit-gandhi · 2022-10-31T16:08:40Z

I'm not sure if Patrick currently has the bandwidth to review this, @sgugger would you be able to take a look if you've got a spare few minutes? Thanks! 🙏

sgugger

Just have some nits on the doc!

sgugger · 2022-11-01T13:04:05Z

-        output = bos_token_ids + token_ids_0
+        ```python
+        >>> tokenizer = WhisperTokenizer.from_pretrained("openai/whisper-tiny", language="spanish")
+        >>> tokenizer.set_prefix_tokens(language="french")  # update the language prefix token


Are we switching from Spanish to Franch here? Would be useful if the comment was clearer on that.

Indeed we are! Resolved in: aa8f4cf

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

Co-authored-by: sgugger <sylvain@huggingface.co>

…tokenizer

patrickvonplaten

Looks good to me, but let's please add one test for set_prefix_tokens e.g. changing the language of the tokenizer on the fly

sanchit-gandhi · 2022-11-02T17:21:04Z

Test for set_prefix_tokens in e98821f

patrickvonplaten · 2022-11-03T12:58:14Z

Cool good to merge for me

ArthurZucker

LGTM thanks a lot

ArthurZucker · 2022-11-03T13:18:54Z


        self.assertListEqual(batch_output, EXPECTED_MULTI)
+
+    def test_set_prefix_tokens(self):


NIce 👍🏻

* [Whisper Tokenizer] Make more user-friendly * use property * make indexing rigorous * small clean-up * tests * skip seq2seq tests * remove multilingual arg * reorder args * collapse to one function Co-authored-by: ArthurZucker <arthur@huggingface.co> * option to override attributes Co-authored-by: ArthurZucker <arthur@huggingface.co> * add to docs * Apply suggestions from code review Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com> * make comment more clear Co-authored-by: sgugger <sylvain@huggingface.co> * don't add special tokens in get_decoder_prompt_ids * add test for set_prefix_tokens Co-authored-by: ArthurZucker <arthur@huggingface.co> Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com> Co-authored-by: sgugger <sylvain@huggingface.co>

[Whisper Tokenizer] Make more user-friendly

37aab85

sanchit-gandhi added 7 commits October 27, 2022 14:02

use property

b507cce

make indexing rigorous

95dc234

small clean-up

0a7f9e9

tests

270295a

skip seq2seq tests

b9b0e98

remove multilingual arg

2c7a9da

reorder args

dcd0403

sanchit-gandhi requested review from ArthurZucker and patrickvonplaten October 27, 2022 18:32

sanchit-gandhi commented Oct 27, 2022

View reviewed changes

ArthurZucker reviewed Oct 27, 2022

View reviewed changes

sanchit-gandhi and others added 3 commits October 28, 2022 09:52

collapse to one function

894f842

Co-authored-by: ArthurZucker <arthur@huggingface.co>

option to override attributes

e9cb546

Co-authored-by: ArthurZucker <arthur@huggingface.co>

add to docs

f6d66e3

sanchit-gandhi requested a review from ArthurZucker October 28, 2022 13:46

sgugger approved these changes Nov 1, 2022

View reviewed changes

sanchit-gandhi mentioned this pull request Nov 2, 2022

[blog] Fine-Tuning Whisper for Multilingual ASR huggingface/blog#591

Merged

sanchit-gandhi and others added 3 commits November 2, 2022 16:13

Apply suggestions from code review

d9b3a14

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

make comment more clear

aa8f4cf

Co-authored-by: sgugger <sylvain@huggingface.co>

Merge remote-tracking branch 'origin/whisper-tokenizer' into whisper-…

f86cdb4

…tokenizer

patrickvonplaten reviewed Nov 2, 2022

View reviewed changes

sanchit-gandhi added 2 commits November 2, 2022 16:41

don't add special tokens in get_decoder_prompt_ids

059aa95

add test for set_prefix_tokens

e98821f

ArthurZucker approved these changes Nov 3, 2022

View reviewed changes

sanchit-gandhi merged commit 06d4880 into huggingface:main Nov 3, 2022

This was referenced Nov 14, 2022

[WHISPER] Update modeling tests #20162

Merged

finetune - whisper needs to add EOS token in the labels huggingface/blog#641

Closed

sanchit-gandhi mentioned this pull request Dec 7, 2022

[Whisper] Fix forced decoder ids #20652

Merged

5 tasks

leoniekerken mentioned this pull request Mar 6, 2023

reproduce whisper embeddings hassonlab/247-pickling#148

Closed


		self.assertListEqual(batch_output, EXPECTED_MULTI)

		def test_set_prefix_tokens(self):

Uh oh!

Conversation

sanchit-gandhi commented Oct 27, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Before submitting

Who can review?

Uh oh!

HuggingFaceDocBuilderDev commented Oct 27, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sanchit-gandhi Oct 27, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sanchit-gandhi Oct 27, 2022

Choose a reason for hiding this comment

Uh oh!

sanchit-gandhi Oct 27, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ArthurZucker left a comment

Choose a reason for hiding this comment

Uh oh!

ArthurZucker Oct 27, 2022

Choose a reason for hiding this comment

Uh oh!

sanchit-gandhi Oct 28, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sanchit-gandhi commented Oct 31, 2022

Uh oh!

sgugger left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sgugger Nov 1, 2022

Choose a reason for hiding this comment

Uh oh!

sanchit-gandhi Nov 2, 2022

Choose a reason for hiding this comment

Uh oh!

patrickvonplaten left a comment

Choose a reason for hiding this comment

Uh oh!

sanchit-gandhi commented Nov 2, 2022

Uh oh!

patrickvonplaten commented Nov 3, 2022

Uh oh!

ArthurZucker left a comment

Choose a reason for hiding this comment

Uh oh!

ArthurZucker Nov 3, 2022

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

sanchit-gandhi commented Oct 27, 2022 •

edited

Loading

HuggingFaceDocBuilderDev commented Oct 27, 2022 •

edited

Loading

sanchit-gandhi Oct 27, 2022 •

edited

Loading

sanchit-gandhi Oct 27, 2022 •

edited

Loading

sanchit-gandhi Oct 28, 2022 •

edited

Loading