354 citations · 607 across the 8 of their papers we have counts for
9 papers
Perception Test: A Diagnostic Benchmark for Multimodal Video Models
Viorica Pătrăucean, Lucas Smaira, Ankush Gupta +21
We propose a novel multimodal video benchmark - the Perception Test - to evaluate the perception and reasoning skills of pre-trained multimodal models (e.g. Flamingo, SeViLA, or GP…
Zorro: the masked multimodal transformer
Adrià Recasens, Jason Lin, Joāo Carreira +8
Attention-based models are appealing for multimodal processing because inputs from multiple modalities can be concatenated and fed to a single backbone network - thus requiring ver…
TAP-Vid: A Benchmark for Tracking Any Point in a Video
Carl Doersch, Ankush Gupta, Larisa Markeeva +6
Generic motion understanding from video involves not only tracking objects, but also perceiving how their surfaces deform and move. This information is useful to make inferences ab…
Towards Learning Universal Audio Representations
Luyu Wang, Pauline Luc, Yan Wu +8
The ability to learn universal audio representations that can solve diverse speech, music, and environment tasks can spur many applications that require general sound content under…
Human-Agent Cooperation in Bridge Bidding
Edward Lockhart, Neil Burch, Nolan Bard +4
We introduce a human-compatible reinforcement-learning approach to a cooperative game, making use of a third-party hand-coded human-compatible bot to generate initial training data…
A Short Note on the Kinetics-700-2020 Human Action Dataset
Lucas Smaira, João Carreira, Eric Noland +3
We describe the 2020 edition of the DeepMind Kinetics human action dataset, which replenishes and extends the Kinetics-700 dataset. In this new version, there are at least 700 vide…
Self-Supervised MultiModal Versatile Networks
Jean-Baptiste Alayrac, Adrià Recasens, Rosalia Schneider +6
Videos are a rich source of multi-modal supervision. In this work, we learn representations using self-supervision by leveraging three modalities naturally present in videos: visua…
Visual Grounding in Video for Unsupervised Word Translation
Gunnar A. Sigurdsson, Jean-Baptiste Alayrac, Aida Nematzadeh +5
There are thousands of actively spoken languages on Earth, but a single visual world. Grounding in this visual world has the potential to bridge the gap between all these languages…
End-to-End Learning of Visual Representations from Uncurated Instructional Videos
Antoine Miech, Jean-Baptiste Alayrac, Lucas Smaira +3
Annotating videos is cumbersome, expensive and not scalable. Yet, many strong video models still rely on manually annotated data. With the recent introduction of the HowTo100M data…