3.7k citations · 11.8k across the 85 of their papers we have counts for
22 papers · 1 filter
Latent Space Communication via K-V Cache Alignment
Lucio M. Dery, Zohar Yahav, Henry Prior +3
Solving increasingly complex problems with large language models (LLMs) necessitates a move beyond individual models and towards multi-model systems that can effectively collaborat…
Communication-Efficient Language Model Training Scales Reliably and Robustly: Scaling Laws for DiLoCo
Zachary Charles, Gabriel Teston, Lucio Dery +5
As we scale to more massive machine learning models, the frequent synchronization demands inherent in data-parallel approaches create significant slowdowns, posing a critical chall…
DiPaCo: Distributed Path Composition
Arthur Douillard, Qixuan Feng, Andrei A. Rusu +7
Progress in machine learning (ML) has been fueled by scaling neural network models. This scaling has been enabled by ever more heroic feats of engineering, necessary for accommodat…
Asynchronous Local-SGD Training for Language Modeling
Bo Liu, Rachita Chhaparia, Arthur Douillard +5
Local stochastic gradient descent (Local-SGD), also referred to as federated averaging, is an approach to distributed optimization where each device performs more than one SGD upda…
DiLoCo: Distributed Low-Communication Training of Language Models
Arthur Douillard, Qixuan Feng, Andrei A. Rusu +6
Large language models (LLM) have become a critical component in many applications of machine learning. However, standard approaches to training LLM require a large number of tightl…
A Data Source for Reasoning Embodied Agents
Jack Lanchantin, Sainbayar Sukhbaatar, Gabriel Synnaeve +3
Recent progress in using machine learning models for reasoning tasks has been driven by novel model architectures, large-scale pre-training protocols, and dedicated reasoning datas…
Learning to Reason and Memorize with Self-Notes
Jack Lanchantin, Shubham Toshniwal, Jason Weston +2
Large language models have been shown to struggle with multi-step reasoning, and do not retain previous reasoning steps for future use. We propose a simple method for solving both…
Hash Layers For Large Sparse Models
Stephen Roller, Sainbayar Sukhbaatar, Arthur Szlam +1
We investigate the training of sparse layers that use different parameters for different inputs based on hashing in large Transformer models. Specifically, we modify the feedforwar…
Not All Memories are Created Equal: Learning to Forget by Expiring
Sainbayar Sukhbaatar, Da Ju, Spencer Poff +4
Attention mechanisms have shown promising results in sequence modeling tasks that require long-term memory. Recent work investigated mechanisms to reduce the computational cost of…
Few-shot Sequence Learning with Transformers
Lajanugen Logeswaran, Ann Lee, Myle Ott +3
Few-shot algorithms aim at learning new tasks provided only a handful of training examples. In this work we investigate few-shot learning in the setting where the data points are s…
Fast Adaptation via Policy-Dynamics Value Functions
Roberta Raileanu, Max Goldstein, Arthur Szlam +1
Standard RL algorithms assume fixed environment dynamics and require a significant amount of interaction to adapt to new environments. We introduce Policy-Dynamics Value Functions…
Real or Fake? Learning to Discriminate Machine from Human Generated Text
Anton Bakhtin, Sam Gross, Myle Ott +3
Energy-based models (EBMs), a.k.a. un-normalized models, have had recent successes in continuous spaces. However, they have not been successfully applied to model text sequences. W…
Learning Goal Embeddings via Self-Play for Hierarchical Reinforcement Learning
Sainbayar Sukhbaatar, Emily Denton, Arthur Szlam +1
In hierarchical reinforcement learning a major challenge is determining appropriate low-level policies. We propose an unsupervised learning scheme, based on asymmetric self-play fr…
Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Play
Sainbayar Sukhbaatar, Zeming Lin, Ilya Kostrikov +3
We describe a simple scheme that allows an agent to learn about its environment in an unsupervised manner. Our scheme pits two versions of the same agent, Alice and Bob, against on…
Transformation-Based Models of Video Sequences
Joost van Amersfoort, Anitha Kannan, Marc'Aurelio Ranzato +3
In this work we propose a simple unsupervised approach for next frame prediction in video. Instead of directly predicting the pixels in a frame given past frames, we predict the tr…
Learning Multiagent Communication with Backpropagation
Sainbayar Sukhbaatar, Arthur Szlam, Rob Fergus
Many tasks in AI require the collaboration of multiple agents. Typically, the communication protocol between agents is manually specified and not altered during training. In this p…
MazeBase: A Sandbox for Learning from Games
Sainbayar Sukhbaatar, Arthur Szlam, Gabriel Synnaeve +2
This paper introduces MazeBase: an environment for simple 2D games, designed as a sandbox for machine learning approaches to reasoning and planning. Within it, we create 10 simple…
Convolutional networks and learning invariant to homogeneous multiplicative scalings
Mark Tygert, Arthur Szlam, Soumith Chintala +3
The conventional classification schemes -- notably multinomial logistic regression -- used in conjunction with convolutional networks (convnets) are classical in statistics, design…
A mathematical motivation for complex-valued convolutional networks
Joan Bruna, Soumith Chintala, Yann LeCun +3
A complex-valued convolutional network (convnet) implements the repeated application of the following composition of three operations, recursively applying the composition to an in…
Video (language) modeling: a baseline for generative models of natural videos
MarcAurelio Ranzato, Arthur Szlam, Joan Bruna +3
We propose a strong baseline model for unsupervised feature learning using video data. By learning to predict missing frames or extrapolate future frames from an input video sequen…
Unsupervised Feature Learning by Deep Sparse Coding
Yunlong He, Koray Kavukcuoglu, Yun Wang +2
In this paper, we propose a new unsupervised feature learning framework, namely Deep Sparse Coding (DeepSC), that extends sparse coding to a multi-layer architecture for visual obj…
Spectral Networks and Locally Connected Networks on Graphs
Joan Bruna, Wojciech Zaremba, Arthur Szlam +1
Convolutional Neural Networks are extremely efficient architectures in image and audio recognition tasks, thanks to their ability to exploit the local translational invariance of s…