Reinforcement and Imitation Learning via Interactive No-Regret Learning

Abstract

Recent work has demonstrated that problems-- particularly imitation learning and structured prediction-- where a learner's predictions influence the input-distribution it is tested on can be naturally addressed by an interactive approach and analyzed using no-regret online learning. These approaches to imitation learning, however, neither require nor benefit from information about the cost of actions. We extend existing results in two directions: first, we develop an interactive imitation learning approach that leverages cost information; second, we extend the technique to address reinforcement learning. The results provide theoretical support to the commonly observed successes of online approximate policy iteration. Our approach suggests a broad new family of algorithms and provides a unifying view of existing techniques for imitation and reinforcement learning.

14 pages. Under review for NIPS 2014 conference

References in corpus (7)

• A Contextual-Bandit Approach to Personalized News Article Recommendation
• Doubly Robust Policy Evaluation and Learning
• Contextual Bandit Algorithms with Supervised Learning Guarantees
• Search-based Structured Prediction
• Stability Conditions for Online Learnability
• Error-Correcting Tournaments
• The Interplay Between Stability and Regret in Online Learning

Cited by in corpus (63)

• An Algorithmic Perspective on Imitation Learning
• Query-Efficient Imitation Learning for End-to-End Autonomous Driving
• Learning to Search Better Than Your Teacher
• Deeply AggreVaTeD: Differentiable Imitation Learning for Sequential Prediction
• Deep Imitative Models for Flexible Inference, Planning, and Control
• Learning Algorithms for Active Learning
• Global overview of Imitation Learning
• Hierarchical Imitation and Reinforcement Learning
• Provably Efficient Imitation Learning from Observation Alone
• AC-Teach: A Bayesian Actor-Critic Method for Policy Learning with an Ensemble of Suboptimal Teachers
• Reinforcement Learning on Web Interfaces Using Workflow-Guided Exploration
• Learning Dexterous Manipulation from Suboptimal Experts
• Learning Belief Representations for Imitation Learning in POMDPs
• SEARNN: Training RNNs with Global-Local Losses
• Learning to Search for Dependencies
• Batch Policy Learning under Constraints
• Learning Heuristic Search via Imitation
• Fully Convolutional Search Heuristic Learning for Rapid Path Planners
• Imitation Learning: Progress, Taxonomies and Challenges
• Riemannian Motion Policy Fusion through Learnable Lyapunov Function Reshaping
• Meta-Learning for Contextual Bandit Exploration
• A Credit Assignment Compiler for Joint Prediction
• Adaptive Approximate Policy Iteration
• Convergence of Value Aggregation for Imitation Learning
• Imitation-Projected Programmatic Reinforcement Learning
• Learning Hierarchical Teaching Policies for Cooperative Agents
• Learning Self-Correctable Policies and Value Functions from Demonstrations with Negative Sampling
• Mitigating Covariate Shift in Imitation Learning via Offline Data Without Great Coverage
• Robust Asymmetric Learning in POMDPs
• Parser for Abstract Meaning Representation using Learning to Search
• Uncertainty-sensitive Learning and Planning with Ensembles
• Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospective Curiosity-Encouraging Imitation Learning
• On Value Functions and the Agent-Environment Boundary
• Optimal Completion Distillation for Sequence Learning
• Generative Hybrid Representations for Activity Forecasting with No-Regret Learning
• Learning by Cheating
• Imitation Learning with Recurrent Neural Networks
• Provable Representation Learning for Imitation Learning via Bi-level Optimization
• Myopic Bayesian Design of Experiments via Posterior Sampling and Probabilistic Programming
• Co-training for Policy Learning
• Recruitment-imitation Mechanism for Evolutionary Reinforcement Learning
• RMP2: A Structured Composable Policy Class for Robot Learning
• MobILE: Model-Based Imitation Learning From Observation Alone
• Comparing Human-Centric and Robot-Centric Sampling for Robot Deep Learning from Demonstrations
• Smooth Imitation Learning via Smooth Costs and Smooth Policies
• Adaptive Information Gathering via Imitation Learning
• Diluted Near-Optimal Expert Demonstrations for Guiding Dialogue Stochastic Policy Optimisation
• Learning to Gather Information via Imitation
• Fixing exposure bias with imitation learning needs powerful oracles
• Active Imitation Learning from Multiple Non-Deterministic Teachers: Formulation, Challenges, and Algorithms
• Reparameterized Variational Divergence Minimization for Stable Imitation
• Hierarchical Variational Imitation Learning of Control Programs
• Transfer Reward Learning for Policy Gradient-Based Text Generation
• Explaining Fast Improvement in Online Imitation Learning
• Learning Neural Parsers with Deterministic Differentiable Imitation Learning
• Concurrent Training Improves the Performance of Behavioral Cloning from Observation
• Exploration in Action Space
• Trajectory-based Learning for Ball-in-Maze Games
• BOOK: Storing Algorithm-Invariant Episodes for Deep Reinforcement Learning
• Ranking Policy Gradient
• EDITOR: an Edit-Based Transformer with Repositioning for Neural Machine Translation with Soft Lexical Constraints
• Imitation Learning via Simultaneous Optimization of Policies and Auxiliary Trajectories
• Multi-Preference Actor Critic