🚀 Feature request

The architecture GPT2ForSequenceClassification was added in #7501 in PyTorch. It would be great to have it in TensorFlow (cf. issues #7622), but it would also be great to have it for other causal models: OpenAI GPT, CTRL, TransfoXL

Currently working on OpenAI GPT: @fmcurti (done)

Below is a list of items to follow to make sure the integration of such an architect

Well, Gumbel Distribution is magical. Basically, given a sequence of K logits, i.e., "\log a_1, \log a_2, ..., \log a_K" and K independent gumbel random variables, i.e., "g_1, g_2, ..., g_K". We have

\argmax_i \log a_i + g_i ~ Categorical({a_i / sum(a)})

This gives you a very simple way to sampl