206 citations · 470 across the 23 of their papers we have counts for
25 papers
Seeing Voices: Generating A-Roll Video from Audio with Mirage
Aditi Sundararaman, Amogh Adishesha, Andrew Jaegle +10
From professional filmmaking to user-generated content, creators and consumers have long recognized that the power of video depends on the harmonious integration of what we hear (t…
SODA: Bottleneck Diffusion Models for Representation Learning
Drew A. Hudson, Daniel Zoran, Mateusz Malinowski +6
We introduce SODA, a self-supervised diffusion model, designed for representation learning. The model incorporates an image encoder, which distills a source view into a compact rep…
Active Acquisition for Multimodal Temporal Data: A Challenging Decision-Making Task
Jannik Kossen, Cătălina Cangea, Eszter Vértes +5
We introduce a challenging decision-making task that we call active acquisition for multimodal temporal data (A2MT). In many real-world scenarios, input features are not readily av…
Where Should I Spend My FLOPS? Efficiency Evaluations of Visual Pre-training Methods
Skanda Koppula, Yazhe Li, Evan Shelhamer +5
Self-supervised methods have achieved remarkable success in transfer learning, often achieving the same or better accuracy than supervised pre-training. Most prior work has done so…
Extended Intelligence
David L Barack, Andrew Jaegle
We argue that intelligence, construed as the disposition to perform tasks successfully, is a property of systems composed of agents and their contexts. This is the thesis of extend…
Transframer: Arbitrary Frame Prediction with Generative Models
Charlie Nash, João Carreira, Jacob Walker +4
We present a general-purpose framework for image modelling and vision tasks based on probabilistic frame prediction. Our approach unifies a broad range of tasks, from image segment…
Object discovery and representation networks
Olivier J. Hénaff, Skanda Koppula, Evan Shelhamer +5
The promise of self-supervised learning (SSL) is to leverage large amounts of unlabeled data to solve complex tasks. While there has been excellent progress with simple, image-leve…
HiP: Hierarchical Perceiver
Joao Carreira, Skanda Koppula, Daniel Zoran +10
General perception systems such as Perceivers can process arbitrary modalities in any combination and are able to handle up to a few hundred thousand inputs. They achieve this gene…
General-purpose, long-context autoregressive modeling with Perceiver AR
Curtis Hawthorne, Andrew Jaegle, Cătălina Cangea +12
Real-world data is high-dimensional: a book, image, or musical performance can easily contain hundreds of thousands of elements even after compression. However, the most commonly u…
SyMetric: Measuring the Quality of Learnt Hamiltonian Dynamics Inferred from Vision
Irina Higgins, Peter Wirnsberger, Andrew Jaegle +1
A recently proposed class of models attempts to learn latent dynamics from high-dimensional observations, like images, using priors informed by Hamiltonian mechanics. While these m…
Which priors matter? Benchmarking models for learning latent dynamics
Aleksandar Botev, Andrew Jaegle, Peter Wirnsberger +2
Learning dynamics is at the heart of many important applications of machine learning (ML), such as robotics and autonomous driving. In these settings, ML algorithms typically need…
Towards Learning Universal Audio Representations
Luyu Wang, Pauline Luc, Yan Wu +8
The ability to learn universal audio representations that can solve diverse speech, music, and environment tasks can spur many applications that require general sound content under…
Imitation by Predicting Observations
Andrew Jaegle, Yury Sulsky, Arun Ahuja +3
Imitation learning enables agents to reuse and adapt the hard-won expertise of others, offering a solution to several key challenges in learning behavior. Although it is easy to ob…
Perceiver IO: A General Architecture for Structured Inputs & Outputs
Andrew Jaegle, Sebastian Borgeaud, Jean-Baptiste Alayrac +12
A central goal of machine learning is the development of systems that can solve many problems in as many data domains as possible. Current architectures, however, cannot be applied…
Perceiver: General Perception with Iterative Attention
Andrew Jaegle, Felix Gimeno, Andrew Brock +3
Biological systems perceive the world by simultaneously processing high-dimensional inputs from modalities as diverse as vision, audition, touch, proprioception, etc. The perceptio…
Game Plan: What AI can do for Football, and What Football can do for AI
Karl Tuyls, Shayegan Omidshafiei, Paul Muller +33
The rapid progress in artificial intelligence (AI) and machine learning has opened unprecedented analytics possibilities in various team and individual sports, including baseball,…
Physically Embedded Planning Problems: New Challenges for Reinforcement Learning
Mehdi Mirza, Andrew Jaegle, Jonathan J. Hunt +9
Recent work in deep reinforcement learning (RL) has produced algorithms capable of mastering challenging games such as Go, chess, or shogi. In these works the RL agent directly obs…
Beyond Tabula-Rasa: a Modular Reinforcement Learning Approach for Physically Embedded 3D Sokoban
Peter Karkus, Mehdi Mirza, Arthur Guez +5
Intelligent robots need to achieve abstract objectives using concrete, spatiotemporally complex sensory information and motor control. Tabula rasa deep reinforcement learning (RL)…
Hamiltonian Generative Networks
Peter Toth, Danilo Jimenez Rezende, Andrew Jaegle +3
The Hamiltonian formalism plays a central role in classical and quantum physics. Hamiltonians are the main tool for modelling the continuous time evolution of systems with conserve…
Keyframing the Future: Keyframe Discovery for Visual Prediction and Planning
Karl Pertsch, Oleh Rybkin, Jingyun Yang +5
Temporal observations such as videos contain essential information about the dynamics of the underlying scene, but they are often interleaved with inessential, predictable details.…
Visual novelty, curiosity, and intrinsic reward in machine learning and the brain
Andrew Jaegle, Vahid Mehrpour, Nicole Rust
A strong preference for novelty emerges in infancy and is prevalent across the animal kingdom. When incorporated into reinforcement-based machine learning algorithms, visual novelt…
Learning what you can do before doing anything
Oleh Rybkin, Karl Pertsch, Konstantinos G. Derpanis +2
Intelligent agents can learn to represent the action spaces of other agents simply by observing them act. Such representations help agents quickly learn to predict the effects of t…
Predicting the Future with Transformational States
Andrew Jaegle, Oleh Rybkin, Konstantinos G. Derpanis +1
An intelligent observer looks at the world and sees not only what is, but what is moving and what can be moved. In other words, the observer sees how the present state of the world…
Understanding image motion with group representations
Andrew Jaegle, Stephen Phillips, Daphne Ippolito +1
Motion is an important signal for agents in dynamic environments, but learning to represent motion from unlabeled video is a difficult and underconstrained problem. We propose a mo…