[Whisper Tokenizer] Make more user-friendly#19921

Merged

sanchit-gandhi merged 16 commits into

huggingface:mainfrom

sanchit-gandhi:whisper-tokenizer

Nov 3, 2022

Commits on Oct 27, 2022

[Whisper Tokenizer] Make more user-friendly
sanchit-gandhi
committed
use property
sanchit-gandhi
committed
make indexing rigorous
sanchit-gandhi
committed
small clean-up
sanchit-gandhi
committed
tests
sanchit-gandhi
committed
skip seq2seq tests
sanchit-gandhi
committed
remove multilingual arg
sanchit-gandhi
committed
reorder args
sanchit-gandhi
committed

Commits on Oct 28, 2022

collapse to one function

sanchit-gandhi
and
ArthurZucker
committed
option to override attributes

sanchit-gandhi
and
ArthurZucker
committed
add to docs
sanchit-gandhi
committed

Commits on Nov 2, 2022

Apply suggestions from code review

sanchit-gandhi
and
sgugger
authored
make comment more clear

sanchit-gandhi
and
sgugger
committed
Merge remote-tracking branch 'origin/whisper-tokenizer' into whisper-tokenizer
sanchit-gandhi
committed
don't add special tokens in get_decoder_prompt_ids
sanchit-gandhi
committed
add test for set_prefix_tokens
sanchit-gandhi
committed