Natural language processing

https://github.com/huggingface/transformers/blob/546dc24e0883e5e9f5eb06ec8060e3e6ccc5f6d7/src/transformers/models/gpt2/modeling_gpt2.py#L698

Assertions can't be relied upon for control flow because they can be disabled, as per the following:

$ python --help
usage: python [option] ... [-c cmd | -m mod | file | -] [arg] ...
...
-O     : remove assert and __debug__-dependent statem

In gensim/models/fasttext.py:

    model = FastText(
        vector_size=m.dim,
        vector_size=m.dim,
        window=m.ws,
        window=m.ws,
        epochs=m.epoch,
        epochs=m.epoch,
        negative=m.neg,
        negative=m.neg,
        # FIXME: these next 2 lines read in unsupported FB FT modes (loss=3 softmax or loss=4 onevsall,
        # or model=3 supervi

The new GrpcDataRuntime (https://github.com/jina-ai/jina/blob/master/jina/peapods/runtimes/grpc/__init__.py) does not support sharding at the moment. Thus a Flow should not specify this runtime and a sharded pod at the same time.

Right now this will not raise any error, but silently produce unexpected behavior. This is undesirable and the Flow should rather check this condition and raise an app

Is your feature request related to a problem? Please describe.
I typically used compressed datasets (e.g. gzipped) to save disk space. This works fine with AllenNLP during training because I can write my dataset reader to load the compressed data. However, the predict command opens the file and reads lines for the Predictor. This fails when it tries to load data from my compressed files.

The below input should remove all user handles which start with "@".

input: @remy:This is waaaaayyyy too much for you!!!!!!@adam
output : [':', 'This', 'is', 'waaayyy', 'too', 'much', 'for', 'you', '!', '!', '!', '@adam']

The TweetTokenizer fail to remove the user handle of Adam.

I would like to open a pull request that solves the following issues:-

• Improve the regular expression