What's Changed
Okay mostly doing the release for these PR:
- Update deserialize of added tokens by @ArthurZucker in #1891
- update stub for typing by @ArthurZucker in #1896
- bump PyO3 to 0.26 by @davidhewitt in #1901
Basically good typing with at least ty, and a lot fast (from 4 to 8x faster) loading vocab with a lot of added tokens and GIL free !?
-
ci: add support for building Win-ARM64 wheels by @MugundanMCW in #1869
-
Add cargo-semver-checks to Rust CI workflow by @haixuanTao in #1875
-
Update indicatif dependency by @gordonmessmer in #1867
-
Bump node-forge from 1.3.1 to 1.3.2 in /tokenizers/examples/unstable_wasm/www by @dependabot[bot] in #1889
-
Bump js-yaml from 3.14.1 to 3.14.2 in /bindings/node by @dependabot[bot] in #1892
-
fix: used normalize_str in BaseTokenizer.normalize by @ishitab02 in #1884
-
Remove runtime stderr warning from Python bindings by @Copilot in #1898
-
Mark immutable pyclasses as frozen by @ngoldbaum in #1861
-
DOCS: add
add_prefix_spacetoprocessors.ByteLevelby @CloseChoice in #1878 -
Bump express from 4.21.2 to 4.22.1 in /tokenizers/examples/unstable_wasm/www by @dependabot[bot] in #1903
New Contributors
- @MugundanMCW made their first contribution in #1869
- @haixuanTao made their first contribution in #1875
- @gordonmessmer made their first contribution in #1867
- @ishitab02 made their first contribution in #1884
- @Copilot made their first contribution in #1898
- @ngoldbaum made their first contribution in #1861
- @CloseChoice made their first contribution in #1878
Full Changelog: v0.22.1...v0.22.2