language-model

Environment info

• transformers version: 4.11.3
• Platform: Linux-4.19.128-microsoft-standard-x86_64-with-glibc2.2.5
• Python version: 3.8.12
• PyTorch version (GPU?): 1.9.1+cu102 (False)
• Tensorflow version (GPU?): not install

From paper, it mentioned

Instead, the training data generator chooses 15% of tokens at random, e.g., in the sentence my
dog is hairy it chooses hairy.

It means that 15% of token will be choose for sure.

From https://github.com/codertimo/BERT-pytorch/blob/master/bert_pytorch/dataset/dataset.py#L68,
for every single token, it has 15% of chance that go though the followup procedure.

We recently created a German Question Answering Dataset and also a German Dense Passage Retrieval dataset, along with trained models for each.

It would be great to have a tutorial (something along the lines of Tutorial 1) that allows users to start playing around with these models!

Issue to track tutorial requests:

• Deep Learning with PyTorch: A 60 Minute Blitz - #69
• Sentence Classification - #79