Reinforcement Learning with Videos: Combining Offline Observations with Interaction

Abstract

Reinforcement learning is a powerful framework for robots to acquire skills from experience, but often requires a substantial amount of online data collection. As a result, it is difficult to collect sufficiently diverse experiences that are needed for robots to generalize broadly. Videos of humans, on the other hand, are a readily available source of broad and interesting experiences. In this paper, we consider the question: can we perform reinforcement learning directly on experience collected by humans? This problem is particularly difficult, as such videos are not annotated with actions and exhibit substantial visual domain shift relative to the robot's embodiment. To address these challenges, we propose a framework for reinforcement learning with videos (RLV). RLV learns a policy and value function using experience collected by humans in combination with data collected by robots. In our experiments, we find that RLV is able to leverage such videos to learn challenging vision-based skills with less than half as many samples as RL methods that learn from scratch.

References in corpus (23)

• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Deep Domain Confusion: Maximizing for Domain Invariance
• Soft Actor-Critic Algorithms and Applications
• Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization
• Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards
• Unsupervised Cross-Domain Image Generation
• Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
• Exploration by Random Network Distillation
• A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Models
• Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning
• Third-Person Imitation Learning
• One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learning
• Generative Adversarial Imitation from Observation
• Efficient Exploration via State Marginal Matching
• AWAC: Accelerating Online Reinforcement Learning with Offline Datasets
• Imitating Latent Policies from Observation
• SQIL: Imitation Learning via Reinforcement Learning with Sparse Rewards
• Multiple Interactions Made Easy (MIME): Large Scale Demonstrations Data for Imitation
• Third-Person Visual Imitation Learning via Decoupled Hierarchical Controller
• Semantic Visual Navigation by Watching YouTube Videos
• Domain Adaptive Imitation Learning
• Estimating Q(s,s') with Deep Deterministic Dynamics Gradients
• Cross-Domain Imitation Learning with a Dual Structure

Cited by in corpus (2)

• A Survey of Zero-shot Generalisation in Deep Reinforcement Learning
• MobILE: Model-Based Imitation Learning From Observation Alone