743 citations · 3k across the 44 of their papers we have counts for
14 papers · 1 filter
Reservoir Transformers
Sheng Shen, Alexei Baevski, Ari S. Morcos +3
We demonstrate that transformers obtain impressive performance even when some of the layers are randomly initialized and never updated. Inspired by old and well-established ideas i…
Are all negatives created equal in contrastive instance discrimination?
Tiffany Tianhui Cai, Jonathan Frankle, David J. Schwab +1
Self-supervised learning has recently begun to rival supervised learning on computer vision tasks. Many of the recent approaches have been based on contrastive instance discriminat…
Towards falsifiable interpretability research
Matthew L. Leavitt, Ari Morcos
Methods for understanding the decisions of and mechanisms underlying deep neural networks (DNNs) typically rely on building intuition by emphasizing sensory or semantic features of…
CURI: A Benchmark for Productive Concept Learning Under Uncertainty
Ramakrishna Vedantam, Arthur Szlam, Maximilian Nickel +2
Humans can learn and reason under substantial uncertainty in a space of infinitely many concepts, including structured relational concepts ("a scene with objects that have the same…
Linking average- and worst-case perturbation robustness via class selectivity and dimensionality
Matthew L. Leavitt, Ari Morcos
Representational sparsity is known to affect robustness to input perturbations in deep neural networks (DNNs), but less is known about how the semantic content of representations a…
Joslim: Joint Widths and Weights Optimization for Slimmable Neural Networks
Ting-Wu Chin, Ari S. Morcos, Diana Marculescu
Slimmable neural networks provide a flexible trade-off front between prediction error and computational requirement (such as the number of floating-point operations or FLOPs) with…
On the relationship between class selectivity, dimensionality, and robustness
Matthew L. Leavitt, Ari S. Morcos
While the relative trade-offs between sparse and distributed representations in deep neural networks (DNNs) are well-studied, less is known about how these trade-offs apply to repr…
Plan2Vec: Unsupervised Representation Learning by Latent Plans
Ge Yang, Amy Zhang, Ari S. Morcos +3
In this paper we introduce plan2vec, an unsupervised representation learning approach that is inspired by reinforcement learning. Plan2vec constructs a weighted graph on an image d…
Analyzing Visual Representations in Embodied Navigation Tasks
Erik Wijmans, Julian Straub, Dhruv Batra +3
Recent advances in deep reinforcement learning require a large amount of training data and generally result in representations that are often over specialized to the target task. I…
Selectivity considered harmful: evaluating the causal impact of class selectivity in DNNs
Matthew L. Leavitt, Ari Morcos
The properties of individual neurons are often analyzed in order to understand the biological and artificial neural networks in which they're embedded. Class selectivity-typically…
Training BatchNorm and Only BatchNorm: On the Expressive Power of Random Features in CNNs
Jonathan Frankle, David J. Schwab, Ari S. Morcos
A wide variety of deep learning techniques from style transfer to multitask learning rely on training affine transformations of features. Most prominent among these is the popular…
Representation Learning Through Latent Canonicalizations
Or Litany, Ari Morcos, Srinath Sridhar +2
We seek to learn a representation on a large annotated data source that generalizes to a target domain using limited new supervision. Many prior approaches to this problem have foc…
The Early Phase of Neural Network Training
Jonathan Frankle, David J. Schwab, Ari S. Morcos
Recent studies have shown that many important aspects of neural network learning take place within the very earliest iterations or epochs of training. For example, sparse, trainabl…
Pruning Convolutional Neural Networks with Self-Supervision
Mathilde Caron, Ari Morcos, Piotr Bojanowski +2
Convolutional neural networks trained without supervision come close to matching performance with supervised pre-training, but sometimes at the cost of an even higher number of par…