fix(tokenization): avoid O(T·N·logN) performance regression in convert_tokens_to_ids#46359
Closed
goingforstudying-ctrl wants to merge 2 commits into
Closed
Conversation
…t_tokens_to_ids Use cached _added_tokens_encoder directly in _convert_token_to_id_with_added_voc instead of the added_tokens_encoder property which re-sorts and rebuilds the dict on every access. Before: O(T · N · logN) where T=sequence length, N=added tokens After: O(T) — only scales with sequence length Fixes huggingface#46315
797f2d0 to
e4e83f0
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #46315
Problem
Since the v5 tokenizer refactor (#40936),
PreTrainedTokenizer._convert_token_to_id_with_added_voclooks tokens up via theadded_tokens_encoderproperty, which re-sorts and rebuilds a dict of all added tokens on every access. This makesconvert_tokens_to_idsO(T·N·logN) instead of O(T).Solution
Use the cached
_added_tokens_encoderdict directly instead of the property in_convert_token_to_id_with_added_voc.Changes
src/transformers/tokenization_python.py: useself._added_tokens_encoderinstead ofself.added_tokens_encoderBenchmark
With 5000 added tokens and 400 tokens in sequence: