VKCOM / YouTokenToMe

Watch 20
Star 657
Fork 43

Code
Issues 20
Pull requests 2
Actions
Projects
Security
Insights

Code
Issues
Pull requests
Actions
Projects
Security
Insights

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Pick a username

Email Address

Password

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

20 Open 17 Closed

Author

Filter by author

Label

Filter by label

Use alt + click/return to exclude labels.

Projects

Filter by project

Milestones

Filter by milestone

Assignee

Filter by who’s assigned

Sort

Sort by

Newest Oldest Most commented Least commented Recently updated Least recently updated

Most reactions

[Question] How to learn joint bpe and vocabulary

#74 opened Sep 8, 2020 by cidrugHug8

Controlling word tokenization

#73 opened Aug 20, 2020 by MexicanMan

How to train with multiple corpus files?

#72 opened Aug 15, 2020 by hccho2

Process killed?

#71 opened Jul 31, 2020 by miggymigz

Add Huggingface tokenizer converter

#70 opened May 15, 2020 by alxmamaev

how to get vocab

#69 opened Apr 30, 2020 by wqfengnlpr

Vocabulary contains underscore multiple times?

#68 opened Apr 20, 2020 by RuABraun

"▁" character can be separated when using BPE-dropout

#67 opened Apr 4, 2020 by TIXFeniks

How to generate vocab.json and merges.txt for YTTM tokenizer?

#66 opened Mar 8, 2020 by nikhilno1

Special Tokens

#65 opened Mar 6, 2020 by martingajek

Adding new terms into pre-trained model vocab | Issue in tokenizing OOV keywords

#63 opened Feb 24, 2020 by spate141

Doesn't consider combining characters.

#59 opened Jan 21, 2020 by IDDT

[Feature] Add ability to manual edit vocabulary (add/remove subwords)

#56 opened Jan 9, 2020 by zabzug-pfpt

Is it possible to wrap the calls in pyspark UDFs?

#53 opened Dec 4, 2019 by AlexGidiotis

[Feature] Add support for training on directories (not only on a single file)

#52 opened Dec 3, 2019 by shkarupa-alex

compilation issues help wanted

#50 opened Nov 20, 2019 by jwijffels

Add an option to predefine special tokens enhancement

#44 opened Nov 15, 2019 by rudvlf0413

Support pickling enhancement

#32 opened Oct 23, 2019 by gooppe

[Feature] Add text normalisation as SentencePiece do enhancement

#16 opened Sep 23, 2019 by keotic

convert from sentencepiece enhancement question

#9 opened Aug 1, 2019 by jwijffels

ProTip! Adding no:label will show everything without a label.