84 citations · 290 across the 36 of their papers we have counts for
38 papers
Regularized DeepIV with Model Selection
Zihao Li, Hui Lan, Vasilis Syrgkanis +2
In this paper, we study nonparametric estimation of instrumental variable (IV) regressions. While recent advancements in machine learning have introduced flexible methods for IV es…
Source Condition Double Robust Inference on Functionals of Inverse Problems
Andrew Bennett, Nathan Kallus, Xiaojie Mao +3
We consider estimation of parameters defined as linear functionals of solutions to linear inverse problems. Any such parameter admits a doubly robust representation that depends on…
Off-Policy Evaluation of Ranking Policies under Diverse User Behavior
Haruka Kiyohara, Masatoshi Uehara, Yusuke Narita +3
Ranking interfaces are everywhere in online platforms. There is thus an ever growing interest in their Off-Policy Evaluation (OPE), aiming towards an accurate performance evaluatio…
Provable Reward-Agnostic Preference-Based Reinforcement Learning
Wenhao Zhan, Masatoshi Uehara, Wen Sun +1
Preference-based Reinforcement Learning (PbRL) is a paradigm in which an RL agent learns to optimize a task using pair-wise preference-based feedback over trajectories, rather than…
Provable Offline Preference-Based Reinforcement Learning
Wenhao Zhan, Masatoshi Uehara, Nathan Kallus +2
In this paper, we investigate the problem of offline Preference-based Reinforcement Learning (PbRL) with human feedback where feedback is available in the form of preference betwee…
Minimax Instrumental Variable Regression and Convergence Guarantees without Identification or Closedness
Andrew Bennett, Nathan Kallus, Xiaojie Mao +3
In this paper, we study nonparametric estimation of instrumental variable (IV) regressions. Recently, many flexible machine learning methods have been developed for instrumental va…
Distributional Offline Policy Evaluation with Predictive Error Guarantees
Runzhe Wu, Masatoshi Uehara, Wen Sun
We study the problem of estimating the distribution of the return of a policy using an offline dataset that is not generated from the policy, i.e., distributional offline policy ev…
Offline Minimax Soft-Q-learning Under Realizability and Partial Coverage
Masatoshi Uehara, Nathan Kallus, Jason D. Lee +1
In offline reinforcement learning (RL) we have no opportunity to explore so we must make assumptions that the data is sufficient to guide picking a good policy, taking the form of…
A Review of Off-Policy Evaluation in Reinforcement Learning
Masatoshi Uehara, Chengchun Shi, Nathan Kallus
Reinforcement learning (RL) is one of the most vibrant research frontiers in machine learning and has been recently applied to solve a number of challenging problems. In this paper…
PAC Reinforcement Learning for Predictive State Representations
Wenhao Zhan, Masatoshi Uehara, Wen Sun +1
In this paper we study online Reinforcement Learning (RL) in partially observable dynamical systems. We focus on the Predictive State Representations (PSRs) model, which is an expr…
Inference on Strongly Identified Functionals of Weakly Identified Functions
Andrew Bennett, Nathan Kallus, Xiaojie Mao +3
In a variety of applications, including nonparametric instrumental variable (NPIV) analysis, proximal causal inference under unmeasured confounding, and missing-not-at-random data…
Future-Dependent Value-Based Off-Policy Evaluation in POMDPs
Masatoshi Uehara, Haruka Kiyohara, Andrew Bennett +5
We study off-policy evaluation (OPE) for partially observable MDPs (POMDPs) with general function approximation. Existing methods such as sequential importance sampling estimators…
Computationally Efficient PAC RL in POMDPs with Latent Determinism and Conditional Embeddings
Masatoshi Uehara, Ayush Sekhari, Jason D. Lee +2
We study reinforcement learning with function approximation for large-scale Partially Observable Markov Decision Processes (POMDPs) where the state space and observation space are…
Provably Efficient Reinforcement Learning in Partially Observable Dynamical Systems
Masatoshi Uehara, Ayush Sekhari, Jason D. Lee +2
We study Reinforcement Learning for partially observable dynamical systems using function approximation. We propose a new \textit{Partially Observable Bilinear Actor-Critic framewo…
Efficient Reinforcement Learning in Block MDPs: A Model-free Representation Learning Approach
Xuezhou Zhang, Yuda Song, Masatoshi Uehara +3
We present BRIEE (Block-structured Representation learning with Interleaved Explore Exploit), an algorithm for efficient reinforcement learning in Markov Decision Processes with bl…
A Minimax Learning Approach to Off-Policy Evaluation in Confounded Partially Observable Markov Decision Processes
Chengchun Shi, Masatoshi Uehara, Jiawei Huang +1
We consider off-policy evaluation (OPE) in Partially Observable Markov Decision Processes (POMDPs), where the evaluation policy depends only on observable variables and the behavio…
Representation Learning for Online and Offline RL in Low-rank MDPs
Masatoshi Uehara, Xuezhou Zhang, Wen Sun
This work studies the question of Representation Learning in RL: how can we learn a compact low-dimensional representation such that on top of the representation we can perform RL…
Pessimistic Model-based Offline Reinforcement Learning under Partial Coverage
Masatoshi Uehara, Wen Sun
We study model-based offline Reinforcement Learning with general function approximation without a full coverage assumption on the offline data distribution. We present an algorithm…
Mitigating Covariate Shift in Imitation Learning via Offline Data Without Great Coverage
Jonathan D. Chang, Masatoshi Uehara, Dhruv Sreenivas +2
This paper studies offline Imitation Learning (IL) where an agent learns to imitate an expert demonstrator without additional online environment interactions. Instead, the learner…
Causal Inference Under Unmeasured Confounding With Negative Controls: A Minimax Learning Approach
Nathan Kallus, Xiaojie Mao, Masatoshi Uehara +1
We study the estimation of causal parameters when not all confounders are observed and instead negative controls are available. Recent work has shown how these can enable identific…
Finite Sample Analysis of Minimax Offline Reinforcement Learning: Completeness, Fast Rates and First-Order Efficiency
Masatoshi Uehara, Masaaki Imaizumi, Nan Jiang +3
We offer a theoretical characterization of off-policy evaluation (OPE) in reinforcement learning using function approximation for marginal importance weights and -functions when…
Fast Rates for the Regret of Offline Reinforcement Learning
Yichun Hu, Nathan Kallus, Masatoshi Uehara
We study the regret of reinforcement learning from offline data generated by a fixed behavior policy in an infinite-horizon discounted Markov decision process (MDP). While existing…
Optimal Off-Policy Evaluation from Multiple Logging Policies
Nathan Kallus, Yuta Saito, Masatoshi Uehara
We study off-policy evaluation (OPE) from multiple logging policies, each generating a dataset of fixed size, i.e., stratified sampling. Previous work noted that in this setting th…
Doubly Robust Off-Policy Value and Gradient Estimation for Deterministic Policies
Nathan Kallus, Masatoshi Uehara
Offline reinforcement learning, wherein one uses off-policy data logged by a fixed behavior policy to evaluate and learn new policies, is crucial in applications where experimentat…