The Value of Information When Deciding What to Learn
arXiv:2110.13973
Abstract
All sequential decision-making agents explore so as to acquire knowledge about a particular target. It is often the responsibility of the agent designer to construct this target which, in rich and complex environments, constitutes a onerous burden; without full knowledge of the environment itself, a designer may forge a sub-optimal learning target that poorly balances the amount of information an agent must acquire to identify the target against the target's associated performance shortfall. While recent work has developed a connection between learning targets and rate-distortion theory to address this challenge and empower agents that decide what to learn in an automated fashion, the proposed algorithm does not optimally tackle the equally important challenge of efficient information acquisition. In this work, building upon the seminal design principle of information-directed sampling (Russo & Van Roy, 2014), we address this shortcoming directly to couple optimal information acquisition with the optimal design of learning targets. Along the way, we offer new insights into learning targets from the literature on rate-distortion theory before turning to empirical results that confirm the value of information when deciding what to learn.
arXiv commentAccepted to Neural Information Processing Systems (NeurIPS) 2021
References in corpus (13)
- Adam: A Method for Stochastic Optimization
- Further Optimal Regret Bounds for Thompson Sampling
- Deep Exploration via Randomized Value Functions
- Ensemble Sampling
- An Information-Theoretic Analysis for Thompson Sampling with Many Actions
- Estimation of the Rate-Distortion Function
- Geometrical interpretation and improvements of the Blahut-Arimoto's algorithm
- Information-Directed Exploration for Deep Reinforcement Learning
- Hypermodels for Exploration
- Asymptotically Optimal Information-Directed Sampling
- Mirror Descent and the Information Ratio
- Reinforcement Learning, Bit by Bit
- First-Order Bayesian Regret Analysis of Thompson Sampling