Word Order Does Matter (And Shuffled Language Models Know It)

Vinit Ravishankar, Mostafa Abdou, Artur Kulmizev +1

Recent studies have shown that language models pretrained and/or fine-tuned on randomly permuted sentences exhibit competitive performance on GLUE, putting into question the import…