Batch Stationary Distribution Estimation

Abstract

We consider the problem of approximating the stationary distribution of an ergodic Markov chain given a set of sampled transitions. Classical simulation-based approaches assume access to the underlying process so that trajectories of sufficient length can be gathered to approximate stationary sampling. Instead, we consider an alternative setting where a fixed set of transitions has been collected beforehand, by a separate, possibly unknown procedure. The goal is still to estimate properties of the stationary distribution, but without additional access to the underlying system. We propose a consistent estimator that is based on recovering a correction ratio function over the given data. In particular, we develop a variational power method (VPM) that provides provably consistent estimates under general conditions. In addition to unifying a number of existing approaches from different subfields, we also find that VPM yields significantly better estimates across a range of problems, including queueing, stochastic differential equations, post-processing MCMC, and off-policy evaluation.

References in corpus (7)

• Breaking the Curse of Dimensionality with Convex Neural Networks
• The Noisy Power Method: A Meta Algorithm with Applications
• Consistent On-Line Off-Policy Evaluation
• Using Options and Covariance Testing for Long Horizon Off-Policy Policy Evaluation
• Learning deep kernels for exponential family densities
• Adversarial Learning of a Sampler Based on an Unnormalized Distribution
• Online Factorization and Partition of Complex Networks From Random Walks

Cited by in corpus (6)

• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• Benchmarks for Deep Off-Policy Evaluation
• Deep Reinforcement and InfoMax Learning
• Inexact iterative numerical linear algebra for neural network-based spectral estimation and rare-event prediction
• Autoregressive Dynamics Models for Offline Policy Evaluation and Optimization
• A maximum-entropy approach to off-policy evaluation in average-reward MDPs