DQN-TAMER: Human-in-the-Loop Reinforcement Learning with Intractable Feedback

Abstract

Exploration has been one of the greatest challenges in reinforcement learning (RL), which is a large obstacle in the application of RL to robotics. Even with state-of-the-art RL algorithms, building a well-learned agent often requires too many trials, mainly due to the difficulty of matching its actions with rewards in the distant future. A remedy for this is to train an agent with real-time feedback from a human observer who immediately gives rewards for some actions. This study tackles a series of challenges for introducing such a human-in-the-loop RL scheme. The first contribution of this work is our experiments with a precisely modeled human observer: binary, delay, stochasticity, unsustainability, and natural reaction. We also propose an RL method called DQN-TAMER, which efficiently uses both human feedback and distant rewards. We find that DQN-TAMER agents outperform their baselines in Maze and Taxi simulated environments. Furthermore, we demonstrate a real-world human-in-the-loop RL application where a camera automatically recognizes a user's facial expressions as feedback to the agent while the agent explores a maze.

References in corpus (4)

• Playing Atari with Deep Reinforcement Learning
• Deep reinforcement learning from human preferences
• RUDDER: Return Decomposition for Delayed Rewards
• Generalizing Skills with Semi-Supervised Reinforcement Learning

Cited by in corpus (10)

• Survey on Machine Learning for Traffic-Driven Service Provisioning in Optical Networks
• A Survey on Interactive Reinforcement Learning: Design Principles and Open Challenges
• Widening the Pipeline in Human-Guided Reinforcement Learning with Explanation and Context-Aware Data Augmentation
• FRESH: Interactive Reward Shaping in High-Dimensional State Spaces using Human Feedback
• Accelerating Reinforcement Learning Agent with EEG-based Implicit Human Feedback
• CrowdTransfer: Enabling Crowd Knowledge Transfer in AIoT Community
• Leveraging Human Guidance for Deep Reinforcement Learning Tasks
• Towards Interactive Reinforcement Learning with Intrinsic Feedback
• Battlesnake Challenge: A Multi-agent Reinforcement Learning Playground with Human-in-the-loop
• MarioMix: Creating Aligned Playstyles for Bots with Interactive Reinforcement Learning