233 citations · 528 across the 30 of their papers we have counts for
34 papers
A Short Note on Evaluating RepNet for Temporal Repetition Counting in Videos
Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson +2
We discuss some consistent issues on how RepNet has been evaluated in various papers. As a way to mitigate these issues, we report RepNet performance results on different datasets,…
OVR: A Dataset for Open Vocabulary Temporal Repetition Counting in Videos
Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson +1
We introduce a dataset of annotations of temporal repetitions in videos. The dataset, OVR (pronounced as over), contains annotations for over 72K videos, with each annotation speci…
FlexCap: Describe Anything in Images in Controllable Detail
Debidatta Dwibedi, Vidhi Jain, Jonathan Tompson +2
We introduce FlexCap, a vision-language model that generates region-specific descriptions of varying lengths. FlexCap is trained to produce length-conditioned captions for input bo…
Genie: Generative Interactive Environments
Jake Bruce, Michael Dennis, Ashley Edwards +22
We introduce Genie, the first generative interactive environment trained in an unsupervised manner from unlabelled Internet videos. The model can be prompted to generate an endless…
Learning from One Continuous Video Stream
João Carreira, Michael King, Viorica Pătrăucean +9
We introduce a framework for online learning from a single continuous video stream -- the way people and animals learn, without mini-batches, data augmentation or shuffling. This p…
RoboTAP: Tracking Arbitrary Points for Few-Shot Visual Imitation
Mel Vecerik, Carl Doersch, Yi Yang +6
For robots to be useful outside labs and specialized factories we need a way to teach them new useful behaviors quickly. Current approaches lack either the generality to onboard ne…
RoboCat: A Self-Improving Generalist Agent for Robotic Manipulation
Konstantinos Bousmalis, Giulia Vezzani, Dushyant Rao +36
The ability to leverage heterogeneous robotic experience from different robots and tasks to quickly master novel skills and embodiments has the potential to transform robot learnin…
TAPIR: Tracking Any Point with per-frame Initialization and temporal Refinement
Carl Doersch, Yi Yang, Mel Vecerik +5
We present a novel model for Tracking Any Point (TAP) that effectively tracks any queried point on any physical surface throughout a video sequence. Our approach employs two stages…
Perception Test: A Diagnostic Benchmark for Multimodal Video Models
Viorica Pătrăucean, Lucas Smaira, Ankush Gupta +21
We propose a novel multimodal video benchmark - the Perception Test - to evaluate the perception and reasoning skills of pre-trained multimodal models (e.g. Flamingo, SeViLA, or GP…
Lossless Adaptation of Pretrained Vision Models For Robotic Manipulation
Mohit Sharma, Claudio Fantacci, Yuxiang Zhou +4
Recent works have shown that large models pretrained on common visual learning tasks can provide useful representations for a wide range of specialized perception problems, as well…
TAP-Vid: A Benchmark for Tracking Any Point in a Video
Carl Doersch, Ankush Gupta, Larisa Markeeva +6
Generic motion understanding from video involves not only tracking objects, but also perceiving how their surfaces deform and move. This information is useful to make inferences ab…
Learning Transferable Motor Skills with Hierarchical Latent Mixture Policies
Dushyant Rao, Fereshteh Sadeghi, Leonard Hasenclever +8
For robots operating in the real world, it is desirable to learn reusable behaviours that can effectively be transferred and adapted to numerous tasks and scenarios. We propose an…
Wish you were here: Hindsight Goal Selection for long-horizon dexterous manipulation
Todor Davchev, Oleg Sushkov, Jean-Baptiste Regli +4
Complex sequential tasks in continuous-control settings often require agents to successfully traverse a set of "narrow passages" in their state space. Solving such tasks with a spa…
With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representations
Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson +2
Self-supervised learning algorithms based on instance discrimination train encoders to be invariant to pre-defined transformations of the same instance. While most methods treat di…
Manipulator-Independent Representations for Visual Imitation
Yuxiang Zhou, Yusuf Aytar, Konstantinos Bousmalis
Imitation learning is an effective tool for robotic learning tasks where specifying a reinforcement learning (RL) reward is not feasible or where the exploration problem is particu…
Learning rich touch representations through cross-modal self-supervision
Martina Zambelli, Yusuf Aytar, Francesco Visin +2
The sense of touch is fundamental in several manipulation tasks, but rarely used in robot manipulation. In this work we tackle the problem of learning rich touch features from cros…
Semi-supervised reward learning for offline reinforcement learning
Ksenia Konyushkova, Konrad Zolna, Yusuf Aytar +4
In offline reinforcement learning (RL) agents are trained using a logged dataset. It appears to be the most natural route to attack real-life applications because in domains such a…
Offline Learning from Demonstrations and Unlabeled Experience
Konrad Zolna, Alexander Novikov, Ksenia Konyushkova +6
Behavior cloning (BC) is often practical for robot learning because it allows a policy to be trained offline without rewards, by supervised learning on expert demonstrations. Howev…
Large-scale multilingual audio visual dubbing
Yi Yang, Brendan Shillingford, Yannis Assael +9
We describe a system for large-scale audiovisual translation and dubbing, which translates videos from one language to another. The source language's speech content is transcribed…
Counting Out Time: Class Agnostic Video Repetition Counting in the Wild
Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson +2
We present an approach for estimating the period with which an action is repeated in a video. The crux of the approach lies in constraining the period prediction module to use temp…
Self-Supervised Sim-to-Real Adaptation for Visual Robotic Manipulation
Rae Jeong, Yusuf Aytar, David Khosid +5
Collecting and automatically obtaining reward signals from real robotic visual data for the purposes of training reinforcement learning algorithms can be quite challenging and time…
Scaling data-driven robotics with reward sketching and batch reinforcement learning
Serkan Cabi, Sergio Gómez Colmenarejo, Alexander Novikov +13
We present a framework for data-driven robotics that makes use of a large dataset of recorded robot experience and scales to several tasks using learned reward functions. We show h…
Temporal Cycle-Consistency Learning
Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson +2
We introduce a self-supervised representation learning method based on the task of temporal alignment between videos. The method trains a network using temporal cycle consistency (…
One-Shot High-Fidelity Imitation: Training Large-Scale Deep Nets with RL
Tom Le Paine, Sergio Gómez Colmenarejo, Ziyu Wang +8
Humans are experts at high-fidelity imitation -- closely mimicking a demonstration, often in one attempt. Humans use this ability to quickly solve a task instance, and to bootstrap…