Consider this code that downloads models and tokenizers to disk and then uses BertTokenizer.from_pretrained to load the tokenizer from disk.

ISSUE: BertTokenizer.from_pretrained() does not seem to be compatible with Python's native pathlib module.

# -*- coding: utf-8 -*-
"""
Created on: 25-04-2020
Author: MacwanJ

ISSUE:

The position embedding in the BERT is not the same as in the transformer. Why not use the form in bert?

Spacy has customizable word level tokenizers with rules for multiple languages. I think porting that to rust would add nicely to this package. Having a customizable uniform word level tokenization across platforms (client web, server) and languages would be beneficial. Currently, idk any clean way or whether it's even possible to write bindings for spacy cython.

Spacy Tokenizer Code

https:

Hi,

First thanks for releasing this, it has been quite helpful.
Would be great if the README page mentioned in software requirements the dependency on pytorch-qrnn (for QRNN-based models). Currently, following the instructions and running one of the standard QRNN models will just throw a ModuleNotFoundError with no instructions. Would be great if there was a prior mention and/or a try/catch w

On home page of website: https://nlp.johnsnowlabs.com/ I read "Full Python, Scala, and Java support"

Unfortunately it's 3 days now I'm trying to use Spark NLP in Java without any success.

• I cannot find Java API (JavaDoc) of the framework.
• not event a single example in Java is available
• I do not know Scala, I do not know how to convert things like:
  val testData = spark.createDataFrame(

It would be great to have instructions on how to train a language model from scratch - not just loading the paper's model.

Hi,
When we try to tokenize the following sentence:

If we use spacy

a = spacy.load('en_core_web_lg')

doc = a("I like the link http://www.idph.iowa.gov/ohds/oral-health-center/coordinator")

list(doc)

We got

[I, like, the, link, http://www.idph.iowa.gov, /, ohds, /, oral, -, health, -, center, /, coordinator]

But if we use the Spacy transformer tokenizer:

I think the filenames in models.sh referred to on lines 4-9 should refer to kaldi-generic-en-tdnn_f-r20190609* which is downloaded on line 3.

When using a Finder with a TfidfRetriever (InMemoryDocumentStore) and default TransformersReader all indices and scores are printed (see line 75 in tfidf.py), and there is no meta-data being inserted into the documents which are returned (line 96). I commented out the print call and added the following line to the Document constructor:

meta={'name':self.document_store.get_document_by_id(