Environment info

• transformers version: 4.11.3
• Platform: Linux-4.19.128-microsoft-standard-x86_64-with-glibc2.2.5
• Python version: 3.8.12
• PyTorch version (GPU?): 1.9.1+cu102 (False)
• Tensorflow version (GPU?): not install

Change tensor.data to tensor.detach() due to
pytorch/pytorch#6990 (comment)
tensor.detach() is more robust than tensor.data.

Is your feature request related to a problem? Please describe.
I typically used compressed datasets (e.g. gzipped) to save disk space. This works fine with AllenNLP during training because I can write my dataset reader to load the compressed data. However, the predict command opens the file and reads lines for the Predictor. This fails when it tries to load data from my compressed files.

Maybe add something like this as a pre or post processing step?

It might make sense to download the emoji list and store it as part of the build, so people do not need to load the emojis module, ...

import emoji
from emoji import unicode_codes
import re


EMOJI_UNICODE = unicode_codes.EMOJI_UNICODE['en']
emojis = sorted(EMOJI_UNICODE.values(), key=len, reverse=True)
print (emojis

Hello spoooopyyy hackers 🎃

This is a Hacktoberfest only issue! 👻

This is also data-sciency!

The Problem

Our English dictionary contains words that aren't English, and does not contain common English words.

Examples of non-common words in the dictionary:

"hlithskjalf",
  "hlorrithi",
  "hlqn",
  "hm",
  "hny",
  "ho",
  "hoactzin",
  "hoactzine