Discriminative Particle Filter Reinforcement Learning for Complex Partial Observations

Abstract

Deep reinforcement learning is successful in decision making for sophisticated games, such as Atari, Go, etc. However, real-world decision making often requires reasoning with partial information extracted from complex visual observations. This paper presents Discriminative Particle Filter Reinforcement Learning (DPFRL), a new reinforcement learning framework for complex partial observations. DPFRL encodes a differentiable particle filter in the neural network policy for explicit reasoning with partial observations over time. The particle filter maintains a belief using learned discriminative update, which is trained end-to-end for decision making. We show that using the discriminative update instead of standard generative models results in significantly improved performance, especially for tasks with complex visual observations, because they circumvent the difficulty of modeling complex observations that are irrelevant to decision making. In addition, to extract features from the particle belief, we propose a new type of belief feature based on the moment generating function. DPFRL outperforms state-of-the-art POMDP RL models in Flickering Atari Games, an existing POMDP RL benchmark, and in Natural Flickering Atari Games, a new, more challenging POMDP RL benchmark introduced in this paper. Further, DPFRL performs well for visual navigation with real-world data in the Habitat environment.

Accepted to ICLR 2020

References in corpus (11)

• Playing Atari with Deep Reinforcement Learning
• DRAW: A Recurrent Neural Network For Image Generation
• A Recurrent Latent Variable Model for Sequential Data
• Learning to Navigate in Complex Environments
• Playing FPS Games with Deep Reinforcement Learning
• Learning to Communicate to Solve Riddles with Deep Distributed Recurrent Q-Networks
• On Improving Deep Reinforcement Learning for POMDPs
• Auto-Encoding Sequential Monte Carlo
• DeepMDP: Learning Continuous Latent Space Models for Representation Learning
• Neural Predictive Belief Representations
• Natural Environment Benchmarks for Reinforcement Learning

Cited by in corpus (7)

• Recurrent Off-policy Baselines for Memory-based Continuous Control
• Differentiable Particle Filtering via Entropy-Regularized Optimal Transport
• Dreaming: Model-based Reinforcement Learning by Latent Imagination without Reconstruction
• Contrastive Variational Reinforcement Learning for Complex Observations
• Variational Marginal Particle Filters
• Beyond Tabula-Rasa: a Modular Reinforcement Learning Approach for Physically Embedded 3D Sokoban
• Online Variational Filtering and Parameter Learning