343 citations · 489 across the 23 of their papers we have counts for
7 papers · 1 filter
Towards out-of-distribution generalization in large-scale astronomical surveys: robust networks learn similar representations
Yash Gondhalekar, Sultan Hassan, Naomi Saphra +1
The generalization of machine learning (ML) models to out-of-distribution (OOD) examples remains a key challenge in extracting information from upcoming astronomical surveys. Inter…
Attribute Diversity Determines the Systematicity Gap in VQA
Ian Berlot-Attwell, Kumar Krishna Agrawal, A. Michael Carrell +2
Although modern neural networks often generalize to new combinations of familiar concepts, the conditions that enable such compositionality have long been an open question. In this…
First Tragedy, then Parse: History Repeats Itself in the New Era of Large Language Models
Naomi Saphra, Eve Fleisig, Kyunghyun Cho +1
Many NLP researchers are experiencing an existential crisis triggered by the astonishing success of ChatGPT and other systems based on large language models (LLMs). After such a di…
TRAM: Bridging Trust Regions and Sharpness Aware Minimization
Tom Sherborne, Naomi Saphra, Pradeep Dasigi +1
Sharpness-aware minimization (SAM) reports improving domain generalization by reducing the loss surface curvature in the parameter space. However, generalization during fine-tuning…
Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMs
Angelica Chen, Ravid Shwartz-Ziv, Kyunghyun Cho +2
Most interpretability research in NLP focuses on understanding the behavior and features of a fully trained model. However, certain insights into model behavior may only be accessi…
Latent State Models of Training Dynamics
Michael Y. Hu, Angelica Chen, Naomi Saphra +1
The impact of randomness on model training is poorly understood. How do differences in data order and initialization actually manifest in the model, such that some training runs ou…
Dynamic Masking Rate Schedules for MLM Pretraining
Zachary Ankner, Naomi Saphra, Davis Blalock +2
Most works on transformers trained with the Masked Language Modeling (MLM) objective use the original BERT model's fixed masking rate of 15%. We propose to instead dynamically sche…