Commit graph

5 commits

Author SHA1 Message Date
Daniël de Kok d2e918cc12 python3Packages.tokenizers: 0.8.1 -> 0.9.2
Changelog:

https://github.com/huggingface/tokenizers/releases/tag/python-v0.9.0
https://github.com/huggingface/tokenizers/releases/tag/python-v0.9.1
https://github.com/huggingface/tokenizers/releases/tag/python-v0.9.2

Changes in the derivation:

* Tokenizers uses a newer version of pyo3 that does not require Rust
  nightly anymore. So, we do not have to use any cheat codes anymore.
* Tokenizers is now a mixed Rust/Python project. The way it is set
  up does not work with Maturin, so switch to setuptools-rust instead.
* Add additional data files needed for tests.
* Use `pytestCheckHook`.
2020-10-28 10:41:27 -07:00
Daniël de Kok 403e2506af python3Packages.tokenizers: 0.8.1rc1 -> 0.8.1 2020-08-05 08:23:11 +02:00
Daniël de Kok b0a6bd4281 python3Packages.tokenizers: fix build
Update the parking_lot dependency to be compatible with recent Rust
versions that replace asm! by llvm_asm!:

https://github.com/Amanieu/parking_lot/pull/223

Fixes #94682
2020-08-05 08:19:01 +02:00
Daniël de Kok 918d5cfb33 python3Packages.tokenizers: 0.8.0 -> 0.8.1.rc1
Changelog:
https://github.com/huggingface/tokenizers/releases/tag/python-v0.8.1.rc1
2020-07-07 07:25:13 +02:00
Daniël de Kok 66d0b2a0d0 python3Packages.tokenizers: init at 0.8.0 2020-07-02 21:53:51 +02:00