7.8k citations · 21.1k across the 144 of their papers we have counts for
17 papers · 1 filter
One-Step Diffusion Distillation via Deep Equilibrium Models
Zhengyang Geng, Ashwini Pokle, J. Zico Kolter
Diffusion models excel at producing high-quality samples but naively require hundreds of iterations, prompting multiple attempts to distill the generation process into a faster net…
Discovering modular solutions that generalize compositionally
Simon Schug, Seijin Kobayashi, Yassir Akram +6
Many complex tasks can be decomposed into simpler, independent parts. Discovering such underlying compositional structure has the potential to enable compositional generalization.…
Continual Learning: Applications and the Road Forward
Eli Verwimp, Rahaf Aljundi, Shai Ben-David +17
Continual learning is a subfield of machine learning, which aims to allow machine learning models to continuously learn on new data, by accumulating knowledge without forgetting wh…
Uncovering mesa-optimization algorithms in Transformers
Johannes von Oswald, Maximilian Schlegel, Alexander Meulemans +10
Some autoregressive models exhibit in-context learning capabilities: being able to learn as an input sequence is processed, without undergoing any parameter changes, and without be…
Towards Robust and Efficient Continual Language Learning
Adam Fisch, Amal Rannen-Triki, Razvan Pascanu +4
As the application space of language models continues to evolve, a natural question to ask is how we can quickly adapt models to new tasks. We approach this classic question from a…
Universality of Linear Recurrences Followed by Non-linear Projections: Finite-Width Guarantees and Benefits of Complex Eigenvalues
Antonio Orvieto, Soham De, Caglar Gulcehre +2
Deep neural networks based on linear RNNs interleaved with position-wise MLPs are gaining traction as competitive approaches for sequence modeling. Examples of such architectures i…
Promoting Exploration in Memory-Augmented Adam using Critical Momenta
Pranshu Malviya, Gonçalo Mordido, Aristide Baratin +5
Adaptive gradient-based optimizers, notably Adam, have left their mark in training large-scale deep learning models, offering fast convergence and robustness to hyperparameter sett…
Latent Space Representations of Neural Algorithmic Reasoners
Vladimir V. Mirjanić, Razvan Pascanu, Petar Veličković
Neural Algorithmic Reasoning (NAR) is a research area focused on designing neural architectures that can reliably capture classical computation, usually by learning to execute algo…
Kalman Filter for Online Classification of Non-Stationary Data
Michalis K. Titsias, Alexandre Galashov, Amal Rannen-Triki +3
In Online Continual Learning (OCL) a learning system receives a stream of data and sequentially performs prediction and training steps. Important challenges in OCL are concerned wi…
Asynchronous Algorithmic Alignment with Cocycles
Andrew Dudzik, Tamara von Glehn, Razvan Pascanu +1
State-of-the-art neural algorithmic reasoners make use of message passing in graph neural networks (GNNs). But typical GNNs blur the distinction between the definition and invocati…
Learning to Modulate pre-trained Models in RL
Thomas Schmied, Markus Hofmarcher, Fabian Paischer +2
Reinforcement Learning (RL) has been successful in various domains like robotics, game playing, and simulation. While RL agents have shown impressive capabilities in their specific…
The Tunnel Effect: Building Data Representations in Deep Neural Networks
Wojciech Masarczyk, Mateusz Ostaszewski, Ehsan Imani +3
Deep neural networks are widely known for their remarkable effectiveness across various tasks, with the consensus that deeper networks implicitly learn more complex data representa…
Deep Reinforcement Learning with Plasticity Injection
Evgenii Nikishin, Junhyuk Oh, Georg Ostrovski +4
A growing body of evidence suggests that neural networks employed in deep reinforcement learning (RL) gradually lose their plasticity, the ability to learn from new data; however,…
Towards Compute-Optimal Transfer Learning
Massimo Caccia, Alexandre Galashov, Arthur Douillard +6
The field of transfer learning is undergoing a significant shift with the introduction of large pretrained models which have demonstrated strong adaptability to a variety of downst…
Resurrecting Recurrent Neural Networks for Long Sequences
Antonio Orvieto, Samuel L Smith, Albert Gu +4
Recurrent Neural Networks (RNNs) offer fast inference on long sequences but are hard to optimize and slow to train. Deep state-space models (SSMs) have recently been shown to perfo…
Understanding plasticity in neural networks
Clare Lyle, Zeyu Zheng, Evgenii Nikishin +3
Plasticity, the ability of a neural network to quickly change its predictions in response to new information, is essential for the adaptability and robustness of deep reinforcement…
SemPPL: Predicting pseudo-labels for better contrastive representations
Matko Bošnjak, Pierre H. Richemond, Nenad Tomasev +7
Learning from large amounts of unsupervised data and a small amount of supervision is an important open problem in computer vision. We propose a new semi-supervised learning method…