Block or Report
Block or report guillaume-be
Report abuse
Contact GitHub support about this user’s behavior. Learn more about reporting abuse.
Report abusePinned
-
rust-tokenizers Public
Rust-tokenizer offers high-performance tokenizers for modern language models, including WordPiece, Byte-Pair Encoding (BPE) and Unigram (SentencePiece) models
-
-
-
huggingface/transformers Public
🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.
258 contributions in the last year
Less
More
Contribution activity
September 2022
Created 3 commits in 2 repositories
Created a pull request in guillaume-be/rust-tokenizers that received 9 comments
Special token map extension
This PR handles the addition of special token maps for all vocabularies and tokenizers. Normalizes the crate interface to special tokens Replaces …
+2,953
−1,321
•
9
comments


