Skip to content

Why GitHub?
Features →
- Customer stories →
- Security →
Team
Enterprise
Explore
- Explore GitHub →
Learn & contribute
Connect with others
Marketplace
Pricing
Plans →
- Compare plans
- Contact Sales
- Nonprofit →
- Education →

In this repository All GitHub

No suggested jump to results

In this repository All GitHub
In this repository All GitHub

Sign in Sign up

huggingface / tokenizers

Watch 64
Star 3.5k
Fork 224

Code
Issues 50
Pull requests 6
Actions
Projects
Security
Insights

More

Code
Issues
Pull requests
Actions
Projects
Security
Insights

Labels 12 Milestones 0

Labels 12 Milestones 0

New pull request New

6 Open 135 Closed

6 Open 135 Closed

Author

Filter by author

Label

Filter by label

Use alt + click/return to exclude labels.

Projects

Filter by project

Milestones

Filter by milestone

Reviews

Filter by reviews

No reviews Review required Approved review Changes requested

Assignee

Filter by who’s assigned

Sort

Sort by

Newest Oldest Most commented Least commented Recently updated Least recently updated

Most reactions

Clean up various normalization methods.

#344 opened Jul 16, 2020 by sebpuetz

remove use of parallel iterators except in batch methods

#308 opened Jun 17, 2020 by epwalsh

3

Port of unigram algorithm.

#292 opened Jun 6, 2020 by Narsil

3

Added Rust code to train from word count (not made available in Python)

#183 opened Mar 4, 2020 by chrhad

Expose train with word counts API

#90 opened Jan 19, 2020 by vilunov

1

TokenizerBuilder

#27 opened Jan 2, 2020 by epwalsh

4

ProTip! Mix and match filters to narrow down what you’re looking for.

© 2020 GitHub, Inc.
Terms
Privacy
Security
Status
Help

Contact GitHub
Pricing
API
Training
Blog
About

You can’t perform that action at this time.

You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.