Listen and Chant Before You Read: The Ladder of Beauty in LM Pre-Training

Yoshinori Nomura

We show that pre-training a Transformer on music before language significantly accelerates language acquisition. Using piano performances (MAESTRO dataset), a developmental pipelin…