A very good first issue IMO!

See huggingface/transformers#4829 (comment)

we should update the eval dataset to pick one start_position (or the most frequent one)

Optionally, use the huggingface/nlp library to get the eval dataset, and hook it into the Trainer.

Also referenced in huggingface/transformers#6997 (comment)

I'm playing around with this wonderful code but I'm running into a curious issue when I try to train the model with my own data.

I replicated the personachat_self_original.json file structure and added my own data. I deleted dataset_cache_OpenAIGPTTokenizer file but when I try to train, I get this error:

INFO:train.py:Pad inputs and convert to Tensor
Traceback (most recent call last)

Ideally, we'd support something like mnli += {pretrain_data_fraction = 0.5}, pretrain_tasks = {mnli,boolq}. Currently, pretrain_data_fraction is just a global argument for all pretraining tasks.

Navigate to: https://www.notion.so/API-Developer-Toolkit-49595ed6ffcd413e93ebff10d7e70fe7
Open a Prompt from the library: https://www.notion.so/Idea-Generator-56b1f1d323a747979ed905bbe0c3782c
Edit dolores/prompts.json, including the prompt text and recommended hyperparameter settings:

"idea-generator": [
{
"prompt": "Here is a list of 100 interesting ideas for new movie plots. Each plo