Transformer Language Models Handle Word Frequency in Prediction Head

Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi +1

Prediction head is a crucial component of Transformer language models. Despite its direct impact on prediction, this component has often been overlooked in analyzing Transformers.…