#
transformers
Here are 3 public repositories matching this topic...
An NLP-suite powered by deep learning
-
Updated
Jun 17, 2020 - Rust
nlp
natural-language-processing
transformers
gpt
language-model
bert
natural-language-understanding
text-tokenization
-
Updated
Jan 15, 2020 - Rust
Improve this page
Add a description, image, and links to the transformers topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the transformers topic, visit your repo's landing page and select "manage topics."
Spacy has customizable word level tokenizers with rules for multiple languages. I think porting that to rust would add nicely to this package. Having a customizable uniform word level tokenization across platforms (client web, server) and languages would be beneficial. Currently, idk any clean way or whether it's even possible to write bindings for spacy cython.
Spacy Tokenizer Code
https: