Smoother Entropy for Active State Trajectory Estimation and Obfuscation in POMDPs

Abstract

We study the problem of controlling a partially observed Markov decision process (POMDP) to either aid or hinder the estimation of its state trajectory. We encode the estimation objectives via the smoother entropy, which is the conditional entropy of the state trajectory given measurements and controls. Consideration of the smoother entropy contrasts with previous approaches that instead resort to marginal (or instantaneous) state entropies due to tractability concerns. By establishing novel expressions for the smoother entropy in terms of the POMDP belief state, we show that both the problems of minimising and maximising the smoother entropy in POMDPs can surprisingly be reformulated as belief-state Markov decision processes with concave cost and value functions. The significance of these reformulations is that they render the smoother entropy a tractable optimisation objective, with structural properties amenable to the use of standard POMDP solution techniques for both active estimation and obfuscation. Simulations illustrate that optimisation of the smoother entropy leads to superior trajectory estimation and obfuscation compared to alternative approaches.

arXiv comment41 pages, 3 figures, accepted for publication in IEEE Transactions on Automatic Control

References in corpus (12)

• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Value-Function Approximations for Partially Observable Markov Decision Processes
• Differentially Private State Estimation in Distribution Networks with Smart Meters
• Real-Time Privacy-Preserving Data Release for Smart Meters
• Active Classification for POMDPs: a Kalman-like State Estimator
• How to Calibrate your Adversary's Capabilities? Inverse Filtering for Counter-Autonomous Systems
• Comprehensive Introduction to Fully Homomorphic Encryption for Dynamic Feedback Controller via LWE-based Cryptosystem
• Inverse Filtering for Hidden Markov Models with Applications to Counter-Adversarial Autonomous Systems
• Entropy Maximization for Markov Decision Processes Under Temporal Logic Constraints
• Nonlinear Fisher Particle Output Feedback Control and its application to Terrain Aided Navigation
• Monte Carlo Information-Oriented Planning
• How to Protect Your Privacy? A Framework for Counter-Adversarial Decision Making

Cited by in corpus (1)

• Entropy-Regularized Partially Observed Markov Decision Processes