743 citations · 3k across the 44 of their papers we have counts for
6 papers · 1 filter
Trade-offs of Local SGD at Scale: An Empirical Study
Jose Javier Gonzalez Ortiz, Jonathan Frankle, Mike Rabbat +2
As datasets and models become increasingly large, distributed training has become a necessary component to allow deep neural networks to train in reasonable amounts of time. Howeve…
Transformed CNNs: recasting pre-trained convolutional layers with self-attention
Stéphane d'Ascoli, Levent Sagun, Giulio Biroli +1
Vision Transformers (ViT) have recently emerged as a powerful alternative to convolutional networks (CNNs). Although hybrid models attempt to bridge the gap between these two archi…
Grounding inductive biases in natural images:invariance stems from variations in data
Diane Bouchacourt, Mark Ibrahim, Ari S. Morcos
To perform well on unseen and potentially out-of-distribution samples, it is desirable for machine learning models to have a predictable response with respect to transformations af…
Width Transfer: On the (In)variance of Width Optimization
Ting-Wu Chin, Diana Marculescu, Ari S. Morcos
Optimizing the channel counts for different layers of a CNN has shown great promise in improving the efficiency of CNNs at test-time. However, these methods often introduce large c…
Characterizing and Improving the Robustness of Self-Supervised Learning through Background Augmentations
Chaitanya K. Ryali, David J. Schwab, Ari S. Morcos
Recent progress in self-supervised learning has demonstrated promising results in multiple visual tasks. An important ingredient in high-performing self-supervised methods is the u…
ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases
Stéphane d'Ascoli, Hugo Touvron, Matthew Leavitt +3
Convolutional architectures have proven extremely successful for vision tasks. Their hard inductive biases enable sample-efficient learning, but come at the cost of a potentially l…