297 citations · 821 across the 15 of their papers we have counts for
18 papers
Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
Gemini Team, Petko Georgiev, Ving Ian Lei +1132
In this report, we introduce the Gemini 1.5 family of models, representing the next generation of highly compute-efficient multimodal models capable of recalling and reasoning over…
TacticAI: an AI assistant for football tactics
Zhe Wang, Petar Veličković, Daniel Hennes +20
Identifying key patterns of tactics implemented by rival teams, and developing effective responses, lies at the heart of modern football. However, doing so algorithmically remains…
Zorro: the masked multimodal transformer
Adrià Recasens, Jason Lin, Joāo Carreira +8
Attention-based models are appealing for multimodal processing because inputs from multiple modalities can be concatenated and fed to a single backbone network - thus requiring ver…
TAP-Vid: A Benchmark for Tracking Any Point in a Video
Carl Doersch, Ankush Gupta, Larisa Markeeva +6
Generic motion understanding from video involves not only tracking objects, but also perceiving how their surfaces deform and move. This information is useful to make inferences ab…
HiP: Hierarchical Perceiver
Joao Carreira, Skanda Koppula, Daniel Zoran +10
General perception systems such as Perceivers can process arbitrary modalities in any combination and are able to handle up to a few hundred thousand inputs. They achieve this gene…
Towards Learning Universal Audio Representations
Luyu Wang, Pauline Luc, Yan Wu +8
The ability to learn universal audio representations that can solve diverse speech, music, and environment tasks can spur many applications that require general sound content under…
Multimodal Self-Supervised Learning of General Audio Representations
Luyu Wang, Pauline Luc, Adria Recasens +2
We present a multimodal framework to learn general audio representations from videos. Existing contrastive audio representation learning methods mainly focus on using the audio mod…
Broaden Your Views for Self-Supervised Video Learning
Adrià Recasens, Pauline Luc, Jean-Baptiste Alayrac +11
Most successful self-supervised learning methods are trained to align the representations of two independent views from the data. State-of-the-art methods in video are inspired by…
A Deep Learning Approach for Characterizing Major Galaxy Mergers
Skanda Koppula, Victor Bapst, Marc Huertas-Company +15
Fine-grained estimation of galaxy merger stages from observations is a key problem useful for validation of our current theoretical understanding of galaxy formation. To this end,…
Game Plan: What AI can do for Football, and What Football can do for AI
Karl Tuyls, Shayegan Omidshafiei, Paul Muller +33
The rapid progress in artificial intelligence (AI) and machine learning has opened unprecedented analytics possibilities in various team and individual sports, including baseball,…
Self-Supervised MultiModal Versatile Networks
Jean-Baptiste Alayrac, Adrià Recasens, Rosalia Schneider +6
Videos are a rich source of multi-modal supervision. In this work, we learn representations using self-supervision by leveraging three modalities naturally present in videos: visua…
Context Based Emotion Recognition using EMOTIC Dataset
Ronak Kosti, Jose M. Alvarez, Adria Recasens +1
In our everyday lives and social interactions we often try to perceive the emotional states of people. There has been a lot of research in providing machines with a similar capacit…
Gaze360: Physically Unconstrained Gaze Estimation in the Wild
Petr Kellnhofer, Adria Recasens, Simon Stent +2
Understanding where people are looking is an informative social cue. In this work, we present Gaze360, a large-scale gaze-tracking dataset and method for robust 3D gaze estimation…
Learning to Zoom: a Saliency-Based Sampling Layer for Neural Networks
Adrià Recasens, Petr Kellnhofer, Simon Stent +2
We introduce a saliency-based distortion layer for convolutional neural networks that helps to improve the spatial sampling of input data for a given task. Our differentiable layer…
Synthetically Trained Icon Proposals for Parsing and Summarizing Infographics
Spandan Madan, Zoya Bylinskii, Matthew Tancik +6
Widely used in news, business, and educational media, infographics are handcrafted to effectively communicate messages about complex and often abstract topics including `ways to co…
Jointly Discovering Visual Objects and Spoken Words from Raw Sensory Input
David Harwath, Adrià Recasens, Dídac Surís +3
In this paper, we explore neural network models that learn to associate segments of spoken audio captions with the semantically relevant portions of natural images that they refer…
Understanding Infographics through Textual and Visual Tag Prediction
Zoya Bylinskii, Sami Alsheikh, Spandan Madan +5
We introduce the problem of visual hashtag discovery for infographics: extracting visual elements from an infographic that are diagnostic of its topic. Given an infographic as inpu…
Following Gaze Across Views
Adrià Recasens, Carl Vondrick, Aditya Khosla +1
Following the gaze of people inside videos is an important signal for understanding people and their actions. In this paper, we present an approach for following gaze across views…