Task-Agnostic Dynamics Priors for Deep Reinforcement Learning

Abstract

While model-based deep reinforcement learning (RL) holds great promise for sample efficiency and generalization, learning an accurate dynamics model is often challenging and requires substantial interaction with the environment. A wide variety of domains have dynamics that share common foundations like the laws of classical mechanics, which are rarely exploited by existing algorithms. In fact, humans continuously acquire and use such dynamics priors to easily adapt to operating in new environments. In this work, we propose an approach to learn task-agnostic dynamics priors from videos and incorporate them into an RL agent. Our method involves pre-training a frame predictor on task-agnostic physics videos to initialize dynamics models (and fine-tune them) for unseen target environments. Our frame prediction architecture, SpatialNet, is designed specifically to capture localized physical phenomena and interactions. Our approach allows for both faster policy learning and convergence to better policies, outperforming competitive approaches on several different environments. We also demonstrate that incorporating this prior allows for more effective transfer between environments.

ICML 2019

References in corpus (16)

• Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
• Playing Atari with Deep Reinforcement Learning
• Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
• Short-Term Forecasting of Passenger Demand under On-Demand Ride Services: A Spatio-Temporal Deep Learning Approach
• A Study on Overfitting in Deep Reinforcement Learning
• Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning
• Schema Networks: Zero-shot Transfer with a Generative Causal Model of Intuitive Physics
• Neural Map: Structured Memory for Deep Reinforcement Learning
• Investigating Human Priors for Playing Video Games
• Deep Successor Reinforcement Learning
• Gotta Learn Fast: A New Benchmark for Generalization in RL
• Eigenoption Discovery through the Deep Successor Representation
• Visual Interaction Networks
• Independently Controllable Factors
• Decoupling Dynamics and Reward for Transfer Learning
• Hashing over Predicted Future Frames for Informed Exploration of Deep Reinforcement Learning

Cited by in corpus (8)

• When to Trust Your Model: Model-Based Policy Optimization
• Modern Machine Learning Tools for Monitoring and Control of Industrial Processes: A Survey
• Entity Abstraction in Visual Model-Based Reinforcement Learning
• Forward Prediction for Physical Reasoning
• Learning Potentials of Quantum Systems using Deep Neural Networks
• Model Based Planning with Energy Based Models
• On the Expressivity of Neural Networks for Deep Reinforcement Learning
• Zero-shot Policy Learning with Spatial Temporal RewardDecomposition on Contingency-aware Observation