127 citations · 519 across the 29 of their papers we have counts for
30 papers
Can Pre-Trained Text-to-Image Models Generate Visual Goals for Reinforcement Learning?
Jialu Gao, Kaizhe Hu, Guowei Xu +1
Pre-trained text-to-image generative models can produce diverse, semantically rich, and realistic images from natural language descriptions. Compared with language, images usually…
RL-ViGen: A Reinforcement Learning Benchmark for Visual Generalization
Zhecheng Yuan, Sizhe Yang, Pu Hua +3
Visual Reinforcement Learning (Visual RL), coupled with high-dimensional observations, has consistently confronted the long-standing challenge of out-of-distribution generalization…
MoVie: Visual Model-Based Policy Adaptation for View Generalization
Sizhe Yang, Yanjie Ze, Huazhe Xu
Visual Reinforcement Learning (RL) agents trained on limited views face significant challenges in generalizing their learned abilities to unseen views. This inherent difficulty is…
CEIL: Generalized Contextual Imitation Learning
Jinxin Liu, Li He, Yachen Kang +3
In this paper, we present \textbf{C}ont\textbf{E}xtual \textbf{I}mitation \textbf{L}earning~(CEIL), a general and broadly applicable algorithm for imitation learning (IL). Inspired…
TACO: Temporal Latent Action-Driven Contrastive Loss for Visual Reinforcement Learning
Ruijie Zheng, Xiyao Wang, Yanchao Sun +5
Despite recent progress in reinforcement learning (RL) from raw pixel data, sample inefficiency continues to present a substantial obstacle. Prior works have attempted to address t…
Seizing Serendipity: Exploiting the Value of Past Success in Off-Policy Actor-Critic
Tianying Ji, Yu Luo, Fuchun Sun +3
Learning high-quality -value functions plays a key role in the success of many modern off-policy deep reinforcement learning (RL) algorithms. Previous works primarily focus on a…
Decision Transformer under Random Frame Dropping
Kaizhe Hu, Ray Chen Zheng, Yang Gao +1
Controlling agents remotely with deep reinforcement learning~(DRL) in the real world is yet to come. One crucial stepping stone is to devise RL algorithms that are robust in the fa…
Is Model Ensemble Necessary? Model-based RL via a Single Model with Lipschitz Regularized Value Function
Ruijie Zheng, Xiyao Wang, Huazhe Xu +1
Probabilistic dynamics model ensemble is widely used in existing model-based reinforcement learning methods as it outperforms a single dynamics model in both asymptotic performance…
Pre-Trained Image Encoder for Generalizable Visual Reinforcement Learning
Zhecheng Yuan, Zhengrong Xue, Bo Yuan +4
Learning generalizable policies that can adapt to unseen environments remains challenging in visual Reinforcement Learning (RL). Existing approaches try to acquire a robust represe…
E-MAPP: Efficient Multi-Agent Reinforcement Learning with Parallel Program Guidance
Can Chang, Ni Mu, Jiajun Wu +2
A critical challenge in multi-agent reinforcement learning(MARL) is for multiple agents to efficiently accomplish complex, long-horizon tasks. The agents often have difficulties in…
Scaling up and Stabilizing Differentiable Planning with Implicit Differentiation
Linfeng Zhao, Huazhe Xu, Lawson L. S. Wong
Differentiable planning promises end-to-end differentiability and adaptivity. However, an issue prevents it from scaling up to larger-scale problems: they need to differentiate thr…
Simple Emergent Action Representations from Multi-Task Policy Training
Pu Hua, Yubei Chen, Huazhe Xu
The low-level sensory and motor signals in deep reinforcement learning, which exist in high-dimensional spaces such as image observations or motor torques, are inherently challengi…
Multi-Person 3D Motion Prediction with Multi-Range Transformers
Jiashun Wang, Huazhe Xu, Medhini Narasimhan +1
We propose a novel framework for multi-person 3D motion trajectory prediction. Our key observation is that a human's action and behaviors may highly depend on the other persons aro…
Plan Better Amid Conservatism: Offline Multi-Agent Reinforcement Learning with Actor Rectification
Ling Pan, Longbo Huang, Tengyu Ma +1
Conservatism has led to significant progress in offline reinforcement learning (RL) where an agent learns from pre-collected datasets. However, as many real-world scenarios involve…
Solving Compositional Reinforcement Learning Problems via Task Reduction
Yunfei Li, Yilin Wu, Huazhe Xu +2
We propose a novel learning paradigm, Self-Imitation via Reduction (SIR), for solving compositional reinforcement learning problems. SIR is based on two core ideas: task reduction…
Discovering Diverse Multi-Agent Strategic Behavior via Reward Randomization
Zhenggang Tang, Chao Yu, Boyuan Chen +6
We propose a simple, general and effective technique, Reward Randomization for discovering diverse strategic policies in complex multi-agent games. Combining reward randomization a…
BeBold: Exploration Beyond the Boundary of Explored Regions
Tianjun Zhang, Huazhe Xu, Xiaolong Wang +4
Efficient exploration under sparse rewards remains a key challenge in deep reinforcement learning. To guide exploration, previous work makes extensive use of intrinsic reward (IR).…
Synthesizing Long-Term 3D Human Motion and Interaction in 3D Scenes
Jiashun Wang, Huazhe Xu, Jingwei Xu +2
Synthesizing 3D human motion plays an important role in many graphics applications as well as understanding human activity. While many efforts have been made on generating realisti…
Multi-Agent Collaboration via Reward Attribution Decomposition
Tianjun Zhang, Huazhe Xu, Xiaolong Wang +4
Recent advances in multi-agent reinforcement learning (MARL) have achieved super-human performance in games like Quake 3 and Dota 2. Unfortunately, these techniques require orders-…
Hierarchical Style-based Networks for Motion Synthesis
Jingwei Xu, Huazhe Xu, Bingbing Ni +3
Generating diverse and natural human motion is one of the long-standing goals for creating intelligent characters in the animated world. In this paper, we propose a self-supervised…
Video Prediction via Example Guidance
Jingwei Xu, Huazhe Xu, Bingbing Ni +2
In video prediction tasks, one major challenge is to capture the multi-modal nature of future contents and dynamics. In this work, we propose a simple yet effective framework that…
Multi-Task Reinforcement Learning with Soft Modularization
Ruihan Yang, Huazhe Xu, Yi Wu +1
Multi-task learning is a very challenging problem in reinforcement learning. While training multiple tasks jointly allow the policies to share parameters across different tasks, th…
Zero-shot Policy Learning with Spatial Temporal RewardDecomposition on Contingency-aware Observation
Huazhe Xu, Boyuan Chen, Yang Gao +1
It is a long-standing challenge to enable an intelligent agent to learn in one environment and generalize to an unseen environment without further data collection and finetuning. I…
Learning Self-Correctable Policies and Value Functions from Demonstrations with Negative Sampling
Yuping Luo, Huazhe Xu, Tengyu Ma
Imitation learning, followed by reinforcement learning algorithms, is a promising paradigm to solve complex control tasks sample-efficiently. However, learning from demonstrations…