What is the Best Sequence Length for BABYLM?

Suchir Salhan, Richard Diehl Martinez, Zébulon Goriely +1

Transformer language models typically operate with a fixed-length context window, which has grown in step with large-scale pretraining datasets. In the BabyLM Challenge, however, m…