Recent Advances in Leveraging Human Guidance for Sequential Decision-Making Tasks

Abstract

A longstanding goal of artificial intelligence is to create artificial agents capable of learning to perform tasks that require sequential decision making. Importantly, while it is the artificial agent that learns and acts, it is still up to humans to specify the particular task to be performed. Classical task-specification approaches typically involve humans providing stationary reward functions or explicit demonstrations of the desired tasks. However, there has recently been a great deal of research energy invested in exploring alternative ways in which humans may guide learning agents that may, e.g., be more suitable for certain tasks or require less human effort. This survey provides a high-level overview of five recent machine learning frameworks that primarily rely on human guidance apart from pre-specified reward functions or conventional, step-by-step action demonstrations. We review the motivation, assumptions, and implementation of each framework, and we discuss possible future research directions.

arXiv commentSpringer journal, Autonomous Agents and Multi-Agent Systems (JAAMAS)

References in corpus (36)

• Deep Visual Domain Adaptation: A Survey
• Dueling Network Architectures for Deep Reinforcement Learning
• Salient Object Detection: A Benchmark
• The Arcade Learning Environment: An Evaluation Platform for General Agents
• DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills
• Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation
• Appearance-Based Gaze Estimation in the Wild
• Human-level performance in first-person multiplayer games with population-based deep reinforcement learning
• A Survey and Critique of Multiagent Deep Reinforcement Learning
• Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization
• Deep reinforcement learning from human preferences
• Cooperative Inverse Reinforcement Learning
• An Algorithmic Perspective on Imitation Learning
• FeUdal Networks for Hierarchical Reinforcement Learning
• Data-Efficient Hierarchical Reinforcement Learning
• Deep Pictorial Gaze Estimation
• Modular Multitask Reinforcement Learning with Policy Sketches
• Trial without Error: Towards Safe Reinforcement Learning via Human Intervention
• Learning Invariant Feature Spaces to Transfer Skills with Reinforcement Learning
• Interactive Learning from Policy-Dependent Human Feedback
• Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning
• Mutual Context Network for Jointly Estimating Egocentric Gaze and Actions
• Reward learning from human preferences and demonstrations in Atari
• Multi-Modal Imitation Learning from Unstructured Demonstrations using Generative Adversarial Nets
• Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations
• Imitation Learning from Observations by Minimizing Inverse Dynamics Disagreement
• The Green Choice: Learning and Influencing Human Decisions on Shared Roads
• Digging Deeper into Egocentric Gaze Prediction
• Provably Efficient Imitation Learning from Observation Alone
• Interactively shaping robot behaviour with unlabeled human instructions
• Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning
• CompILE: Compositional Imitation Learning and Execution
• DDCO: Discovery of Deep Continuous Options for Robot Learning from Demonstrations
• Adversarial Imitation via Variational Inverse Reinforcement Learning
• Model-based Behavioral Cloning with Future Image Similarity Learning
• Preference learning along multiple criteria: A game-theoretic perspective