93 citations · 446 across the 32 of their papers we have counts for
8 papers · 1 filter
Lasting Effects of Abstract Pretraining Beyond Perplexity
Zachary Shinnick, Hemanth Saratchandran, Damien Teney +1
Language models are typically pretrained from random initialization. Recent work challenges this convention, showing that a brief warm-up on abstract, algorithmically generated dat…
Procedural Core: A Compact Recurrent Initialization for Vision Transformers
Zachary Shinnick, Christian Internò, Hemanth Saratchandran +2
Transformers are typically trained from random initialization, requiring all their capabilities to emerge from large-scale optimization. Recent work showed that a small amount of a…
Transformers Pretrained on Procedural Data Contain Modular Structures for Algorithmic Reasoning
Zachary Shinnick, Liangze Jiang, Hemanth Saratchandran +2
Pretraining on large, semantically rich datasets is key for developing language models. Surprisingly, recent studies have shown that even synthetic data, generated procedurally thr…
Leaner Transformers: More Heads, Less Depth
Hemanth Saratchandran, Damien Teney, Simon Lucey
Transformers have reshaped machine learning by utilizing attention mechanisms to capture complex patterns in large datasets, leading to significant improvements in performance. Thi…
Selective Mixup Helps with Distribution Shifts, But Not (Only) because of Mixup
Damien Teney, Jindong Wang, Ehsan Abbasnejad
Mixup is a highly successful technique to improve generalization of neural networks by augmenting the training data with combinations of random pairs. Selective mixup is a family o…
ID and OOD Performance Are Sometimes Inversely Correlated on Real-world Datasets
Damien Teney, Yong Lin, Seong Joon Oh +1
Several studies have compared the in-distribution (ID) and out-of-distribution (OOD) performance of models in computer vision and NLP. They report a frequent positive correlation a…
Predicting is not Understanding: Recognizing and Addressing Underspecification in Machine Learning
Damien Teney, Maxime Peyrard, Ehsan Abbasnejad
Machine learning (ML) models are typically optimized for their accuracy on a given dataset. However, this predictive criterion rarely captures all desirable properties of a model,…
Evading the Simplicity Bias: Training a Diverse Set of Models Discovers Solutions with Superior OOD Generalization
Damien Teney, Ehsan Abbasnejad, Simon Lucey +1
Neural networks trained with SGD were recently shown to rely preferentially on linearly-predictive features and can ignore complex, equally-predictive ones. This simplicity bias ca…