Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora

Abstract

Children can acquire language from less than 100 million words of input. Large language models are far less data-efficient: they typically require 3 or 4 orders of magnitude more data and still do not perform as well as humans on many evaluations. These intensive resource demands limit the ability of researchers to train new models and use existing models as developmentally plausible cognitive models. The BabyLM Challenge is a communal effort in which participants compete to optimize language model training on a fixed data budget. Submissions are compared on various evaluation tasks targeting grammatical ability, downstream task performance, and generalization. Participants can submit to up to three tracks with progressively looser data restrictions. From over 30 submissions, we extract concrete recommendations on how best to train data-efficient language models, and on where future efforts should (and perhaps should not) focus. The winning submissions using the LTG-BERT architecture (Samuel et al., 2023) outperformed models trained on trillions of words. Other submissions achieved strong results through training on shorter input sequences or training a student model on a pretrained teacher. Curriculum learning attempts, which accounted for a large number of submissions, were largely unsuccessful, though some showed modest improvements.

Published in Proceedings of BabyLM. Please cite the published version on ACL anthology: http://aclanthology.org/2023.conll-babylm.1/

References in corpus (25)

• Decoupled Weight Decay Regularization
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• Scaling Laws for Neural Language Models
• Dialogue Act Modeling for Automatic Tagging and Recognition of Conversational Speech
• Emergent Abilities of Large Language Models
• SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
• The Goldilocks Principle: Reading Children's Books with Explicit Memory Representations
• Cognitive Science in the era of Artificial Intelligence: A roadmap for reverse-engineering the infant language-learner
• DataPerf: Benchmarks for Data-Centric AI Development
• minicons: Enabling Flexible Behavioral and Representational Analyses of Transformer Language Models
• NormFormer: Improved Transformer Pretraining with Extra Normalization
• Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Models
• Inverse Scaling: When Bigger Isn't Better
• TIES-Merging: Resolving Interference When Merging Models
• Human-Adversarial Visual Question Answering
• Call for Papers -- The BabyLM Challenge: Sample-efficient pretraining on a developmentally plausible corpus
• Modular Deep Learning
• ReLoRA: High-Rank Training Through Low-Rank Updates
• FairyTailor: A Multimodal Generative Framework for Storytelling
• Git-Theta: A Git Extension for Collaborative Development of Machine Learning Models
• The MiniPile Challenge for Data-Efficient Language Models
• Where to start? Analyzing the potential value of intermediate models
• Mini Minds: Exploring Bebeshka and Zlata Baby Models
• Lil-Bevo: Explorations of Strategies for Training Language Models in More Humanlike Ways
• Increasing The Performance of Cognitively Inspired Data-Efficient Language Models via Implicit Structure Building

Cited by in corpus (4)

• Natural Language Processing RELIES on Linguistics
• Mini Minds: Exploring Bebeshka and Zlata Baby Models
• Increasing The Performance of Cognitively Inspired Data-Efficient Language Models via Implicit Structure Building
• Lil-Bevo: Explorations of Strategies for Training Language Models in More Humanlike Ways