2.9k citations · 4.3k across the 28 of their papers we have counts for
32 papers · 1 filter
TAPIR: Tracking Any Point with per-frame Initialization and temporal Refinement
Carl Doersch, Yi Yang, Mel Vecerik +5
We present a novel model for Tracking Any Point (TAP) that effectively tracks any queried point on any physical surface throughout a video sequence. Our approach employs two stages…
Perception Test: A Diagnostic Benchmark for Multimodal Video Models
Viorica Pătrăucean, Lucas Smaira, Ankush Gupta +21
We propose a novel multimodal video benchmark - the Perception Test - to evaluate the perception and reasoning skills of pre-trained multimodal models (e.g. Flamingo, SeViLA, or GP…
Zorro: the masked multimodal transformer
Adrià Recasens, Jason Lin, Joāo Carreira +8
Attention-based models are appealing for multimodal processing because inputs from multiple modalities can be concatenated and fed to a single backbone network - thus requiring ver…
TAP-Vid: A Benchmark for Tracking Any Point in a Video
Carl Doersch, Ankush Gupta, Larisa Markeeva +6
Generic motion understanding from video involves not only tracking objects, but also perceiving how their surfaces deform and move. This information is useful to make inferences ab…
Where Should I Spend My FLOPS? Efficiency Evaluations of Visual Pre-training Methods
Skanda Koppula, Yazhe Li, Evan Shelhamer +5
Self-supervised methods have achieved remarkable success in transfer learning, often achieving the same or better accuracy than supervised pre-training. Most prior work has done so…
Transframer: Arbitrary Frame Prediction with Generative Models
Charlie Nash, João Carreira, Jacob Walker +4
We present a general-purpose framework for image modelling and vision tasks based on probabilistic frame prediction. Our approach unifies a broad range of tasks, from image segment…
Object discovery and representation networks
Olivier J. Hénaff, Skanda Koppula, Evan Shelhamer +5
The promise of self-supervised learning (SSL) is to leverage large amounts of unlabeled data to solve complex tasks. While there has been excellent progress with simple, image-leve…
HiP: Hierarchical Perceiver
Joao Carreira, Skanda Koppula, Daniel Zoran +10
General perception systems such as Perceivers can process arbitrary modalities in any combination and are able to handle up to a few hundred thousand inputs. They achieve this gene…
Input-level Inductive Biases for 3D Reconstruction
Wang Yifan, Carl Doersch, Relja Arandjelović +2
Much of the recent progress in 3D vision has been driven by the development of specialized architectures that incorporate geometrical inductive biases. In this paper we tackle 3D r…
Efficient Visual Pretraining with Contrastive Detection
Olivier J. Hénaff, Skanda Koppula, Jean-Baptiste Alayrac +3
Self-supervised pretraining has been shown to yield powerful representations for transfer learning. These performance gains come at a large computational cost however, with state-o…
Perceiver: General Perception with Iterative Attention
Andrew Jaegle, Felix Gimeno, Andrew Brock +3
Biological systems perceive the world by simultaneously processing high-dimensional inputs from modalities as diverse as vision, audition, touch, proprioception, etc. The perceptio…
A Short Note on the Kinetics-700-2020 Human Action Dataset
Lucas Smaira, João Carreira, Eric Noland +3
We describe the 2020 edition of the DeepMind Kinetics human action dataset, which replenishes and extends the Kinetics-700 dataset. In this new version, there are at least 700 vide…
The AVA-Kinetics Localized Human Actions Video Dataset
Ang Li, Meghana Thotakuri, David A. Ross +3
This paper describes the AVA-Kinetics localized human actions video dataset. The dataset is collected by annotating videos from the Kinetics-700 dataset using the AVA annotation pr…
Visual Grounding in Video for Unsupervised Word Translation
Gunnar A. Sigurdsson, Jean-Baptiste Alayrac, Aida Nematzadeh +5
There are thousands of actively spoken languages on Earth, but a single visual world. Grounding in this visual world has the potential to bridge the gap between all these languages…
Controllable Attention for Structured Layered Video Decomposition
Jean-Baptiste Alayrac, João Carreira, Relja Arandjelović +1
The objective of this paper is to be able to separate a video into its natural layers, and to control which of the separated layers to attend to. For example, to be able to separat…
A Short Note on the Kinetics-700 Human Action Dataset
Joao Carreira, Eric Noland, Chloe Hillier +1
We describe an extension of the DeepMind Kinetics human action dataset from 600 classes to 700 classes, where for each class there are at least 600 video clips from different YouTu…
Video Action Transformer Network
Rohit Girdhar, João Carreira, Carl Doersch +1
We introduce the Action Transformer model for recognizing and localizing human actions in video clips. We repurpose a Transformer-style architecture to aggregate features from the…
The Visual Centrifuge: Model-Free Layered Video Representations
Jean-Baptiste Alayrac, João Carreira, Andrew Zisserman
True video understanding requires making sense of non-lambertian scenes where the color of light arriving at the camera sensor encodes information about not just the last object it…
A Short Note about Kinetics-600
Joao Carreira, Eric Noland, Andras Banki-Horvath +2
We describe an extension of the DeepMind Kinetics human action dataset from 400 classes, each with at least 400 video clips, to 600 classes, each with at least 600 video clips. In…
A Better Baseline for AVA
Rohit Girdhar, João Carreira, Carl Doersch +1
We introduce a simple baseline for action localization on the AVA dataset. The model builds upon the Faster R-CNN bounding box detection framework, adapted to operate on pure spati…
Massively Parallel Video Networks
Joao Carreira, Viorica Patraucean, Laurent Mazare +2
We introduce a class of causal video understanding models that aims to improve efficiency of video processing by maximising throughput, minimising latency, and reducing the number…
The Kinetics Human Action Video Dataset
Will Kay, Joao Carreira, Karen Simonyan +9
We describe the DeepMind Kinetics human action video dataset. The dataset contains 400 human action classes, with at least 400 video clips for each action. Each clip lasts around 1…
Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset
Joao Carreira, Andrew Zisserman
The paucity of videos in current action classification datasets (UCF-101 and HMDB-51) has made it difficult to identify good video architectures, as most methods obtain similar per…
Shape and Symmetry Induction for 3D Objects
Shubham Tulsiani, Abhishek Kar, Qixing Huang +2
Actions as simple as grasping an object or navigating around it require a rich understanding of that object's 3D shape from a given viewpoint. In this paper we repurpose powerful l…