Policy Information Capacity: Information-Theoretic Measure for Task Complexity in Deep Reinforcement Learning

Abstract

Progress in deep reinforcement learning (RL) research is largely enabled by benchmark task environments. However, analyzing the nature of those environments is often overlooked. In particular, we still do not have agreeable ways to measure the difficulty or solvability of a task, given that each has fundamentally different actions, observations, dynamics, rewards, and can be tackled with diverse RL algorithms. In this work, we propose policy information capacity (PIC) -- the mutual information between policy parameters and episodic return -- and policy-optimal information capacity (POIC) -- between policy parameters and episodic optimality -- as two environment-agnostic, algorithm-agnostic quantitative metrics for task difficulty. Evaluating our metrics across toy environments as well as continuous control benchmark tasks from OpenAI Gym and DeepMind Control Suite, we empirically demonstrate that these information-theoretic metrics have higher correlations with normalized task solvability scores than a variety of alternatives. Lastly, we show that these metrics can also be used for fast and compute-efficient optimizations of key design parameters such as reward shaping, policy architectures, and MDP properties for better solvability by RL algorithms without ever running full RL experiments.

arXiv commentAccepted to ICML2021. The code is available at: https://github.com/frt03/pic

References in corpus (26)

• Continuous control with deep reinforcement learning
• Trust Region Policy Optimization
• Soft Actor-Critic Algorithms and Applications
• Evolution Strategies as a Scalable Alternative to Reinforcement Learning
• QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
• Don't Decay the Learning Rate, Increase the Batch Size
• Rainbow: Combining Improvements in Deep Reinforcement Learning
• Reinforcement Learning with Deep Energy-Based Policies
• Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
• D4RL: Datasets for Deep Data-Driven Reinforcement Learning
• Continuous Deep Q-Learning with Model-based Acceleration
• Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning
• Maximum a Posteriori Policy Optimisation
• Provably Efficient Reinforcement Learning with Linear Function Approximation
• Variational Intrinsic Control
• Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning
• Temporal Difference Models: Model-Free Deep RL for Model-Based Control
• On Variational Bounds of Mutual Information
• Contextual Decision Processes with Low Bellman Rank are PAC-Learnable
• Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Critic
• Sample Complexity of Episodic Fixed-Horizon Reinforcement Learning
• Unsupervised Control Through Non-Parametric Discriminative Rewards
• Reinforcement Learning with General Value Function Approximation: Provably Efficient Approach via Bounded Eluder Dimension
• Action and Perception as Divergence Minimization
• On Oracle-Efficient PAC RL with Rich Observations
• A Unified Bellman Optimality Principle Combining Reward Maximization and Empowerment