d2e918cc12
Changelog: https://github.com/huggingface/tokenizers/releases/tag/python-v0.9.0 https://github.com/huggingface/tokenizers/releases/tag/python-v0.9.1 https://github.com/huggingface/tokenizers/releases/tag/python-v0.9.2 Changes in the derivation: * Tokenizers uses a newer version of pyo3 that does not require Rust nightly anymore. So, we do not have to use any cheat codes anymore. * Tokenizers is now a mixed Rust/Python project. The way it is set up does not work with Maturin, so switch to setuptools-rust instead. * Add additional data files needed for tests. * Use `pytestCheckHook`. |
||
---|---|---|
.. | ||
androidndk-pkgs | ||
arduino | ||
beam-modules | ||
bower-modules/generic | ||
chez-modules | ||
compilers | ||
coq-modules | ||
dhall-modules | ||
dotnet-modules | ||
em-modules/generic | ||
go-modules | ||
go-packages | ||
guile-modules | ||
haskell-modules | ||
idris-modules | ||
interpreters | ||
java-modules | ||
libraries | ||
lisp-modules | ||
lua-modules | ||
misc | ||
mobile | ||
node-packages | ||
ocaml-modules | ||
perl-modules | ||
pharo | ||
php-packages | ||
pure-modules | ||
python-modules | ||
r-modules | ||
ruby-modules | ||
tools | ||
web |