On Improving Deep Reinforcement Learning for POMDPs

Abstract

Deep Reinforcement Learning (RL) recently emerged as one of the most competitive approaches for learning in sequential decision making problems with fully observable environments, e.g., computer Go. However, very little work has been done in deep RL to handle partially observable environments. We propose a new architecture called Action-specific Deep Recurrent Q-Network (ADRQN) to enhance learning performance in partially observable domains. Actions are encoded by a fully connected layer and coupled with a convolutional observation to form an action-observation pair. The time series of action-observation pairs are then integrated by an LSTM layer that learns latent states based on which a fully connected layer computes Q-values as in conventional Deep Q-Networks (DQNs). We demonstrate the effectiveness of our new architecture in several partially observable domains, including flickering Atari games.

7 pages, 6 figures, 3 tables

References in corpus (1)

• Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models

Cited by in corpus (19)

• Deep Reinforcement Learning for Multi-Agent Systems: A Review of Challenges, Solutions and Applications
• Federated Reinforcement Learning: Techniques, Applications, and Open Challenges
• Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Model
• A Bayesian Framework for Digital Twin-Based Control, Monitoring, and Data Collection in Wireless Systems
• DeepTraffic: Crowdsourced Hyperparameter Tuning of Deep Reinforcement Learning Systems for Multi-Agent Dense Traffic Navigation
• Discriminative Particle Filter Reinforcement Learning for Complex Partial Observations
• Approximate information state for approximate planning and reinforcement learning in partially observed systems
• Recurrent Off-policy Baselines for Memory-based Continuous Control
• Deep Reinforcement Learning for Autonomous Internet of Things: Model, Applications and Challenges
• Robust Reinforcement Learning in POMDPs with Incomplete and Noisy Observations
• Optimizing Sequential Medical Treatments with Auto-Encoding Heuristic Search in POMDPs
• DualSMC: Tunneling Differentiable Filtering and Planning under Continuous POMDPs
• Coach-Player Multi-Agent Reinforcement Learning for Dynamic Team Composition
• Efficient LSTM Training with Eligibility Traces
• Learning What to Memorize: Using Intrinsic Motivation to Form Useful Memory in Partially Observable Reinforcement Learning
• Influence-aware Memory Architectures for Deep Reinforcement Learning
• Inductive-bias-driven Reinforcement Learning For Efficient Schedules in Heterogeneous Clusters
• Automatically Learning Fallback Strategies with Model-Free Reinforcement Learning in Safety-Critical Driving Scenarios
• How Are Learned Perception-Based Controllers Impacted by the Limits of Robust Control?