Skip to content
#

pytorch-model

Here are 7 public repositories matching this topic...

lamthuy
lamthuy commented Apr 1, 2020

Hi,
When we try to tokenize the following sentence:

If we use spacy

a = spacy.load('en_core_web_lg')

doc = a("I like the link http://www.idph.iowa.gov/ohds/oral-health-center/coordinator")

list(doc)

We got

[I, like, the, link, http://www.idph.iowa.gov, /, ohds, /, oral, -, health, -, center, /, coordinator]

But if we use the Spacy transformer tokenizer:

Improve this page

Add a description, image, and links to the pytorch-model topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with the pytorch-model topic, visit your repo's landing page and select "manage topics."

Learn more

You can’t perform that action at this time.