145 citations · 279 across the 13 of their papers we have counts for
13 papers
Training Agents Inside of Scalable World Models
Danijar Hafner, Wilson Yan, Timothy Lillicrap
World models learn general knowledge from videos and simulate experience for training behaviors in imagination, offering a path towards intelligent agents. However, previous world…
ElasticTok: Adaptive Tokenization for Image and Video
Wilson Yan, Volodymyr Mnih, Aleksandra Faust +3
Efficient video tokenization remains a key bottleneck in learning general purpose vision models that are capable of processing long video sequences. Prevailing approaches are restr…
World Model on Million-Length Video And Language With Blockwise RingAttention
Hao Liu, Wilson Yan, Matei Zaharia +1
Enabling long-context understanding remains a key challenge in scaling existing sequence models -- a crucial component in developing generally intelligent models that can process a…
Motion-Conditioned Image Animation for Video Editing
Wilson Yan, Andrew Brown, Pieter Abbeel +2
We introduce MoCA, a Motion-Conditioned Image Animation approach for video editing. It leverages a simple decomposition of the video editing problem into image editing followed by…
ALP: Action-Aware Embodied Learning for Perception
Xinran Liang, Anthony Han, Wilson Yan +2
Current methods in training and benchmarking vision models exhibit an over-reliance on passive, curated datasets. Although models trained on these datasets have shown strong perfor…
Video Prediction Models as Rewards for Reinforcement Learning
Alejandro Escontrela, Ademi Adeniji, Wilson Yan +6
Specifying reward signals that allow agents to learn complex behaviors is a long-standing challenge in reinforcement learning. A promising approach is to extract preferences for be…
Language Quantized AutoEncoders: Towards Unsupervised Text-Image Alignment
Hao Liu, Wilson Yan, Pieter Abbeel
Recent progress in scaling up large language models has shown impressive capabilities in performing few-shot learning across a wide range of text-based tasks. However, a key limita…
Temporally Consistent Transformers for Video Generation
Wilson Yan, Danijar Hafner, Stephen James +1
To generate accurate videos, algorithms have to understand the spatial and temporal dependencies in the world. Current algorithms enable accurate predictions over short horizons bu…
Patch-based Object-centric Transformers for Efficient Video Generation
Wilson Yan, Ryo Okumura, Stephen James +1
In this work, we present Patch-based Object-centric Video Transformer (POVT), a novel region-based video generation architecture that leverages object-centric information to effici…
VideoGPT: Video Generation using VQ-VAE and Transformers
Wilson Yan, Yunzhi Zhang, Pieter Abbeel +1
We present VideoGPT: a conceptually simple architecture for scaling likelihood based generative modeling to natural videos. VideoGPT uses VQ-VAE that learns downsampled discrete la…
Learning Predictive Representations for Deformable Objects Using Contrastive Estimation
Wilson Yan, Ashwin Vangipuram, Pieter Abbeel +1
Using visual model-based learning for deformable object manipulation is challenging due to difficulties in learning plannable visual representations along with complex dynamic mode…
Natural Image Manipulation for Autoregressive Models Using Fisher Scores
Wilson Yan, Jonathan Ho, Pieter Abbeel
Deep autoregressive models are one of the most powerful models that exist today which achieve state-of-the-art bits per dim. However, they lie at a strict disadvantage when it come…
Learning to Manipulate Deformable Objects without Demonstrations
Yilin Wu, Wilson Yan, Thanard Kurutach +2
In this paper we tackle the problem of deformable object manipulation through model-free visual reinforcement learning (RL). In order to circumvent the sample inefficiency of RL, w…