Few-Shot Goal Inference for Visuomotor Learning and Planning

Abstract

Reinforcement learning and planning methods require an objective or reward function that encodes the desired behavior. Yet, in practice, there is a wide range of scenarios where an objective is difficult to provide programmatically, such as tasks with visual observations involving unknown object positions or deformable objects. In these cases, prior methods use engineered problem-specific solutions, e.g., by instrumenting the environment with additional sensors to measure a proxy for the objective. Such solutions require a significant engineering effort on a per-task basis, and make it impractical for robots to continuously learn complex skills outside of laboratory settings. We aim to find a more general and scalable solution for specifying goals for robot learning in unconstrained environments. To that end, we formulate the few-shot objective learning problem, where the goal is to learn a task objective from only a few example images of successful end states for that task. We propose a simple solution to this problem: meta-learn a classifier that can recognize new goals from a few examples. We show how this approach can be used with both model-free reinforcement learning and visual model-based planning and show results in three domains: rope manipulation from images in simulation, visual navigation in a simulated 3D environment, and object arrangement into user-specified configurations on a real robot.

arXiv commentVideos available at https://sites.google.com/view/few-shot-goals

References in corpus (18)

• Adam: A Method for Stochastic Optimization
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Theoretical Models of Learning to Learn
• Matching Networks for One Shot Learning
• Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization
• RL${}^2$: Fast Reinforcement Learning via Slow Reinforcement Learning
• Deep reinforcement learning from human preferences
• Learning to reinforcement learn
• Generative Adversarial Imitation Learning
• One-Shot Visual Imitation Learning via Meta-Learning
• Collective Robot Reinforcement Learning with Distributed Asynchronous Guided Policy Search
• Self-Supervised Visual Planning with Temporal Skip Connections
• One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learning
• Meta-Learning and Universality: Deep Representations and Gradient Descent can Approximate any Learning Algorithm
• Universal Planning Networks
• Multi-task Maximum Entropy Inverse Reinforcement Learning
• Learning a Prior over Intent via Meta-Inverse Reinforcement Learning
• Cross-Domain Perceptual Reward Functions

Cited by in corpus (15)

• How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learned
• Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control
• A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms
• Learning to Generalize from Sparse and Underspecified Rewards
• Robot eye-hand coordination learning by watching human demonstrations: a task function approximation approach
• Task-agnostic Exploration in Reinforcement Learning
• Improving Generalization in Meta-learning via Task Augmentation
• Guiding Policies with Language via Meta-Learning
• Example-Driven Model-Based Reinforcement Learning for Solving Long-Horizon Visuomotor Tasks
• VisuoSpatial Foresight for Physical Sequential Fabric Manipulation
• Lifelong Robotic Reinforcement Learning by Retaining Experiences
• Weakly-Supervised Reinforcement Learning for Controllable Behavior
• Replacing Rewards with Examples: Example-Based Policy Search via Recursive Classification
• Gap-Dependent Unsupervised Exploration for Reinforcement Learning
• MELD: Meta-Reinforcement Learning from Images via Latent State Models