Memory-based control with recurrent neural networks
arXiv:1512.04455
Abstract
Partially observed control problems are a challenging aspect of reinforcement learning. We extend two related, model-free algorithms for continuous control -- deterministic policy gradient and stochastic value gradient -- to solve partially observed domains using recurrent neural networks trained with backpropagation through time. We demonstrate that this approach, coupled with long-short term memory is able to solve a variety of physical control problems exhibiting an assortment of memory requirements. These include the short-term integration of information from noisy sensors and the identification of system parameters, as well as long-term memory problems that require preserving information over many time steps. We also demonstrate success on a combined exploration and memory problem in the form of a simplified version of the well-known Morris water maze task. Finally, we show that our approach can deal with high-dimensional observations by learning directly from pixels. We find that recurrent deterministic and stochastic policies are able to learn similarly good solutions to these tasks, including the water maze where the agent must learn effective search strategies.
arXiv commentNIPS Deep Reinforcement Learning Workshop 2015
References in corpus (8)
- Adam: A Method for Stochastic Optimization
- Auto-Encoding Variational Bayes
- Continuous control with deep reinforcement learning
- Trust Region Policy Optimization
- End-to-End Training of Deep Visuomotor Policies
- Learning Continuous Control Policies by Stochastic Value Gradients
- Learning Policies with External Memory
- Compatible Value Gradients for Reinforcement Learning of Continuous Deep Policies
Cited by in corpus (58)
- Continuous control with deep reinforcement learning
- A Brief Survey of Deep Reinforcement Learning
- Deep Reinforcement Learning for Multi-Agent Systems: A Review of Challenges, Solutions and Applications
- Benchmarking Deep Reinforcement Learning for Continuous Control
- Federated Reinforcement Learning: Techniques, Applications, and Open Challenges
- Autonomous Unmanned Aerial Vehicle Navigation using Reinforcement Learning: A Systematic Review
- Review: Deep Learning in Electron Microscopy
- Guided Deep Reinforcement Learning for Swarm Systems
- Model-based Deep Reinforcement Learning for Dynamic Portfolio Optimization
- Learning Implicit Credit Assignment for Cooperative Multi-Agent Reinforcement Learning
- The Free Energy Principle for Perception and Action: A Deep Learning Perspective
- Hybrid Tensor Decomposition in Neural Network Compression
- Sim2Real View Invariant Visual Servoing by Recurrent Control
- Preparing for the Unknown: Learning a Universal Policy with Online System Identification
- Recurrent Deterministic Policy Gradient Method for Bipedal Locomotion on Rough Terrain Challenge
- The Evolution of Reinforcement Learning in Quantitative Finance: A Survey
- Cross-Modal Contrastive Learning of Representations for Navigation using Lightweight, Low-Cost Millimeter Wave Radar for Adverse Environmental Conditions
- Optimal Scheduling in IoT-Driven Smart Isolated Microgrids Based on Deep Reinforcement Learning
- Consequentialist conditional cooperation in social dilemmas with imperfect information
- Variational Recurrent Models for Solving Partially Observable Control Tasks
- SafeCritic: Collision-Aware Trajectory Prediction
- A Survey on Autonomous Vehicle Control in the Era of Mixed-Autonomy: From Physics-Based to AI-Guided Driving Policy Learning
- Deep Reinforcement Learning and Transportation Research: A Comprehensive Review
- Approximate information state for approximate planning and reinforcement learning in partially observed systems
- Deep Reinforcement Learning for Safe Landing Site Selection with Concurrent Consideration of Divert Maneuvers
- Recurrent Off-policy Baselines for Memory-based Continuous Control
- Deep Reinforcement Learning for Autonomous Internet of Things: Model, Applications and Challenges
- Deep Reinforcement Learning for Motion Planning of Mobile Robots
- Map-based Multi-Policy Reinforcement Learning: Enhancing Adaptability of Robots by Deep Reinforcement Learning
- A Deep Reinforcement Learning Framework for Eco-driving in Connected and Automated Hybrid Electric Vehicles
- Actor-critic versus direct policy search: a comparison based on sample complexity
- OffWorld Gym: open-access physical robotics environment for real-world reinforcement learning benchmark and research
- Formal Methods for Autonomous Systems
- Dynamic Control of a Fiber Manufacturing Process using Deep Reinforcement Learning
- A Survey of Exploration Methods in Reinforcement Learning
- Imitating Driver Behavior with Generative Adversarial Networks
- Neural Network Memory Architectures for Autonomous Robot Navigation
- Self-organization of action hierarchy and compositionality by reinforcement learning with recurrent neural networks
- Reinforcement Learning With Temporal Logic Rewards
- Unsupervised Visual Attention and Invariance for Reinforcement Learning
- MELD: Meta-Reinforcement Learning from Images via Latent State Models
- Target-Based Temporal Difference Learning
- Reinforcement Mechanism Design for e-commerce
- Beyond Tabula-Rasa: a Modular Reinforcement Learning Approach for Physically Embedded 3D Sokoban
- Quick Learner Automated Vehicle Adapting its Roadmanship to Varying Traffic Cultures with Meta Reinforcement Learning
- Semi-Decentralized Network Slicing for Reliable V2V Service Provisioning: A Model-free Deep Reinforcement Learning Approach
- Context-Based Soft Actor Critic for Environments with Non-stationary Dynamics
- Analysis of a Target-Based Actor-Critic Algorithm with Linear Function Approximation
- Hierarchical RNNs-Based Transformers MADDPG for Mixed Cooperative-Competitive Environments
- Inducing Functions through Reinforcement Learning without Task Specification
- Graph Convolutional Memory using Topological Priors
- Learning to Actively Reduce Memory Requirements for Robot Control Tasks
- Learning Memory-Dependent Continuous Control from Demonstrations
- Simulation Studies on Deep Reinforcement Learning for Building Control with Human Interaction
- Robust Reinforcement Learning under model misspecification
- Reinforcement Learning using Guided Observability
- Crowdfunding Dynamics Tracking: A Reinforcement Learning Approach
- Implementing Inductive bias for different navigation tasks through diverse RNN attractors