297 citations · 821 across the 15 of their papers we have counts for
12 papers · 1 filter
Zorro: the masked multimodal transformer
Adrià Recasens, Jason Lin, Joāo Carreira +8
Attention-based models are appealing for multimodal processing because inputs from multiple modalities can be concatenated and fed to a single backbone network - thus requiring ver…
TAP-Vid: A Benchmark for Tracking Any Point in a Video
Carl Doersch, Ankush Gupta, Larisa Markeeva +6
Generic motion understanding from video involves not only tracking objects, but also perceiving how their surfaces deform and move. This information is useful to make inferences ab…
HiP: Hierarchical Perceiver
Joao Carreira, Skanda Koppula, Daniel Zoran +10
General perception systems such as Perceivers can process arbitrary modalities in any combination and are able to handle up to a few hundred thousand inputs. They achieve this gene…
Broaden Your Views for Self-Supervised Video Learning
Adrià Recasens, Pauline Luc, Jean-Baptiste Alayrac +11
Most successful self-supervised learning methods are trained to align the representations of two independent views from the data. State-of-the-art methods in video are inspired by…
Self-Supervised MultiModal Versatile Networks
Jean-Baptiste Alayrac, Adrià Recasens, Rosalia Schneider +6
Videos are a rich source of multi-modal supervision. In this work, we learn representations using self-supervision by leveraging three modalities naturally present in videos: visua…
Context Based Emotion Recognition using EMOTIC Dataset
Ronak Kosti, Jose M. Alvarez, Adria Recasens +1
In our everyday lives and social interactions we often try to perceive the emotional states of people. There has been a lot of research in providing machines with a similar capacit…
Gaze360: Physically Unconstrained Gaze Estimation in the Wild
Petr Kellnhofer, Adria Recasens, Simon Stent +2
Understanding where people are looking is an informative social cue. In this work, we present Gaze360, a large-scale gaze-tracking dataset and method for robust 3D gaze estimation…
Learning to Zoom: a Saliency-Based Sampling Layer for Neural Networks
Adrià Recasens, Petr Kellnhofer, Simon Stent +2
We introduce a saliency-based distortion layer for convolutional neural networks that helps to improve the spatial sampling of input data for a given task. Our differentiable layer…
Synthetically Trained Icon Proposals for Parsing and Summarizing Infographics
Spandan Madan, Zoya Bylinskii, Matthew Tancik +6
Widely used in news, business, and educational media, infographics are handcrafted to effectively communicate messages about complex and often abstract topics including `ways to co…
Jointly Discovering Visual Objects and Spoken Words from Raw Sensory Input
David Harwath, Adrià Recasens, Dídac Surís +3
In this paper, we explore neural network models that learn to associate segments of spoken audio captions with the semantically relevant portions of natural images that they refer…
Understanding Infographics through Textual and Visual Tag Prediction
Zoya Bylinskii, Sami Alsheikh, Spandan Madan +5
We introduce the problem of visual hashtag discovery for infographics: extracting visual elements from an infographic that are diagnostic of its topic. Given an infographic as inpu…
Following Gaze Across Views
Adrià Recasens, Carl Vondrick, Aditya Khosla +1
Following the gaze of people inside videos is an important signal for understanding people and their actions. In this paper, we present an approach for following gaze across views…