2.9k citations · 4.3k across the 28 of their papers we have counts for
36 papers
TAPIR: Tracking Any Point with per-frame Initialization and temporal Refinement
Carl Doersch, Yi Yang, Mel Vecerik +5
We present a novel model for Tracking Any Point (TAP) that effectively tracks any queried point on any physical surface throughout a video sequence. Our approach employs two stages…
Perception Test: A Diagnostic Benchmark for Multimodal Video Models
Viorica Pătrăucean, Lucas Smaira, Ankush Gupta +21
We propose a novel multimodal video benchmark - the Perception Test - to evaluate the perception and reasoning skills of pre-trained multimodal models (e.g. Flamingo, SeViLA, or GP…
Zorro: the masked multimodal transformer
Adrià Recasens, Jason Lin, Joāo Carreira +8
Attention-based models are appealing for multimodal processing because inputs from multiple modalities can be concatenated and fed to a single backbone network - thus requiring ver…
TAP-Vid: A Benchmark for Tracking Any Point in a Video
Carl Doersch, Ankush Gupta, Larisa Markeeva +6
Generic motion understanding from video involves not only tracking objects, but also perceiving how their surfaces deform and move. This information is useful to make inferences ab…
Where Should I Spend My FLOPS? Efficiency Evaluations of Visual Pre-training Methods
Skanda Koppula, Yazhe Li, Evan Shelhamer +5
Self-supervised methods have achieved remarkable success in transfer learning, often achieving the same or better accuracy than supervised pre-training. Most prior work has done so…
Transframer: Arbitrary Frame Prediction with Generative Models
Charlie Nash, João Carreira, Jacob Walker +4
We present a general-purpose framework for image modelling and vision tasks based on probabilistic frame prediction. Our approach unifies a broad range of tasks, from image segment…
Object discovery and representation networks
Olivier J. Hénaff, Skanda Koppula, Evan Shelhamer +5
The promise of self-supervised learning (SSL) is to leverage large amounts of unlabeled data to solve complex tasks. While there has been excellent progress with simple, image-leve…
HiP: Hierarchical Perceiver
Joao Carreira, Skanda Koppula, Daniel Zoran +10
General perception systems such as Perceivers can process arbitrary modalities in any combination and are able to handle up to a few hundred thousand inputs. They achieve this gene…
General-purpose, long-context autoregressive modeling with Perceiver AR
Curtis Hawthorne, Andrew Jaegle, Cătălina Cangea +12
Real-world data is high-dimensional: a book, image, or musical performance can easily contain hundreds of thousands of elements even after compression. However, the most commonly u…
Input-level Inductive Biases for 3D Reconstruction
Wang Yifan, Carl Doersch, Relja Arandjelović +2
Much of the recent progress in 3D vision has been driven by the development of specialized architectures that incorporate geometrical inductive biases. In this paper we tackle 3D r…
Towards Learning Universal Audio Representations
Luyu Wang, Pauline Luc, Yan Wu +8
The ability to learn universal audio representations that can solve diverse speech, music, and environment tasks can spur many applications that require general sound content under…
Perceiver IO: A General Architecture for Structured Inputs & Outputs
Andrew Jaegle, Sebastian Borgeaud, Jean-Baptiste Alayrac +12
A central goal of machine learning is the development of systems that can solve many problems in as many data domains as possible. Current architectures, however, cannot be applied…
Efficient Visual Pretraining with Contrastive Detection
Olivier J. Hénaff, Skanda Koppula, Jean-Baptiste Alayrac +3
Self-supervised pretraining has been shown to yield powerful representations for transfer learning. These performance gains come at a large computational cost however, with state-o…
Perceiver: General Perception with Iterative Attention
Andrew Jaegle, Felix Gimeno, Andrew Brock +3
Biological systems perceive the world by simultaneously processing high-dimensional inputs from modalities as diverse as vision, audition, touch, proprioception, etc. The perceptio…
A Short Note on the Kinetics-700-2020 Human Action Dataset
Lucas Smaira, João Carreira, Eric Noland +3
We describe the 2020 edition of the DeepMind Kinetics human action dataset, which replenishes and extends the Kinetics-700 dataset. In this new version, there are at least 700 vide…
The AVA-Kinetics Localized Human Actions Video Dataset
Ang Li, Meghana Thotakuri, David A. Ross +3
This paper describes the AVA-Kinetics localized human actions video dataset. The dataset is collected by annotating videos from the Kinetics-700 dataset using the AVA annotation pr…
Visual Grounding in Video for Unsupervised Word Translation
Gunnar A. Sigurdsson, Jean-Baptiste Alayrac, Aida Nematzadeh +5
There are thousands of actively spoken languages on Earth, but a single visual world. Grounding in this visual world has the potential to bridge the gap between all these languages…
Sideways: Depth-Parallel Training of Video Models
Mateusz Malinowski, Grzegorz Swirszcz, Joao Carreira +1
We propose Sideways, an approximate backpropagation scheme for training video models. In standard backpropagation, the gradients and activations at every computation step through t…
Controllable Attention for Structured Layered Video Decomposition
Jean-Baptiste Alayrac, João Carreira, Relja Arandjelović +1
The objective of this paper is to be able to separate a video into its natural layers, and to control which of the separated layers to attend to. For example, to be able to separat…
A Short Note on the Kinetics-700 Human Action Dataset
Joao Carreira, Eric Noland, Chloe Hillier +1
We describe an extension of the DeepMind Kinetics human action dataset from 600 classes to 700 classes, where for each class there are at least 600 video clips from different YouTu…
Video Action Transformer Network
Rohit Girdhar, João Carreira, Carl Doersch +1
We introduce the Action Transformer model for recognizing and localizing human actions in video clips. We repurpose a Transformer-style architecture to aggregate features from the…
The Visual Centrifuge: Model-Free Layered Video Representations
Jean-Baptiste Alayrac, João Carreira, Andrew Zisserman
True video understanding requires making sense of non-lambertian scenes where the color of light arriving at the camera sensor encodes information about not just the last object it…
A Short Note about Kinetics-600
Joao Carreira, Eric Noland, Andras Banki-Horvath +2
We describe an extension of the DeepMind Kinetics human action dataset from 400 classes, each with at least 400 video clips, to 600 classes, each with at least 600 video clips. In…
A Better Baseline for AVA
Rohit Girdhar, João Carreira, Carl Doersch +1
We introduce a simple baseline for action localization on the AVA dataset. The model builds upon the Faster R-CNN bounding box detection framework, adapted to operate on pure spati…