Privacy-Constrained Policies via Mutual Information Regularized Policy Gradients
arXiv:2012.15019
Abstract
As reinforcement learning techniques are increasingly applied to real-world decision problems, attention has turned to how these algorithms use potentially sensitive information. We consider the task of training a policy that maximizes reward while minimizing disclosure of certain sensitive state variables through the actions. We give examples of how this setting covers real-world problems in privacy for sequential decision-making. We solve this problem in the policy gradients framework by introducing a regularizer based on the mutual information (MI) between the sensitive state and the actions. We develop a model-based stochastic gradient estimator for optimization of privacy-constrained policies. We also discuss an alternative MI regularizer that serves as an upper bound to our main MI regularizer and can be optimized in a model-free setting, and a powerful direct estimator that can be used in an environment with differentiable dynamics. We contrast previous work in differentially-private RL to our mutual-information formulation of information disclosure. Experimental results show that our training method results in policies that hide the sensitive state, even in challenging high-dimensional tasks.
Accepted to AISTATS 2024
References in corpus (12)
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
- Model-Based Reinforcement Learning for Atari
- Safe Exploration in Continuous Action Spaces
- Horizon: Facebook's Open Source Applied Reinforcement Learning Platform
- Learning Controllable Fair Representations
- Differentially Private Contextual Linear Bandits
- The Price of Differential Privacy For Online Learning
- InfoBot: Transfer and Exploration via the Information Bottleneck
- Learning Optimal Fair Policies
- Is a Good Representation Sufficient for Sample Efficient Reinforcement Learning?
- Differential Privacy for Multi-armed Bandits: What Is It and What Is Its Cost?
- Privacy-preserving Q-Learning with Functional Noise in Continuous State Spaces