Large-scale cloze evaluation reveals that token prediction tasks are neither lexically nor semantically aligned

Cassandra L. Jacobs, Loïc Grobol, Alvin Tsang

In this work we compare the generative behavior at the next token prediction level in several language models by comparing them to human productions in the cloze task. We find that…