148 citations · 972 across the 92 of their papers we have counts for
45 papers · 1 filter
A positive resolution of the gap-entropy conjecture
P. M. Aronow, Nathan Kallus, Patrick Lopatto
We prove the gap-entropy conjecture for fixed-confidence best-arm identification with independent unit-variance Gaussian arms, means in , and a unique optimal arm. For each…
Causal Inference on Networks under Misspecified Exposure Mappings: A Partial Identification Framework
Maresa Schröder, Miruna Oprescu, Stefan Feuerriegel +1
Estimating treatment effects in networks is challenging, as each potential outcome depends on the treatments of all other nodes in the network. To overcome this difficulty, existin…
Exploration in the Limit
Brian M. Cho, Nathan Kallus
In fixed-confidence best arm identification (BAI), the objective is to quickly identify the optimal option while controlling the probability of error below a desired threshold. Des…
Entropy After </Think> for reasoning model early exiting
Xi Wang, James McInerney, Lequn Wang +1
Reasoning LLMs show improved performance with longer chains of thought. However, recent work has highlighted their tendency to overthink, continuing to revise answers even after re…
Optimization of Epsilon-Greedy Exploration
Ethan Che, Hakan Ceylan, James McInerney +1
Modern recommendation systems rely on exploration to learn user preferences for new items, typically implementing uniform exploration policies (e.g., epsilon-greedy) due to their s…
Value-Guided Search for Efficient Chain-of-Thought Reasoning
Kaiwen Wang, Jin Peng Zhou, Jonathan Chang +4
In this paper, we propose a simple and efficient method for value model training on long-context reasoning traces. Compared to existing process reward models (PRMs), our method doe…
: Provably Optimal Distributional RL for LLM Post-Training
Jin Peng Zhou, Kaiwen Wang, Jonathan Chang +5
Reinforcement learning (RL) post-training is crucial for LLM alignment and reasoning, but existing policy-based methods, such as PPO and DPO, can fall short of fixing shortcuts inh…
GST-UNet: A Neural Framework for Spatiotemporal Causal Inference with Time-Varying Confounding
Miruna Oprescu, David K. Park, Xihaier Luo +2
Estimating causal effects from spatiotemporal observational data is essential in public health, environmental science, and policy evaluation, where randomized experiments are often…
Reward Maximization for Pure Exploration: Minimax Optimal Good Arm Identification for Nonparametric Multi-Armed Bandits
Brian Cho, Dominik Meier, Kyra Gan +1
In multi-armed bandits, the tasks of reward maximization and pure exploration are often at odds with each other. The former focuses on exploiting arms with the highest means, while…
A Reductions Approach to Risk-Sensitive Reinforcement Learning with Optimized Certainty Equivalents
Kaiwen Wang, Dawen Liang, Nathan Kallus +1
We study risk-sensitive RL where the goal is learn a history-dependent policy that optimizes some risk measure of cumulative rewards. We consider a family of risks called the optim…
Switching the Loss Reduces the Cost in Batch (Offline) Reinforcement Learning
Alex Ayoub, Kaiwen Wang, Vincent Liu +5
We propose training fitted Q-iteration with log-loss (FQI-log) for batch reinforcement learning (RL). We show that the number of samples needed to learn a near-optimal policy with…
More Benefits of Being Distributional: Second-Order Bounds for Reinforcement Learning
Kaiwen Wang, Owen Oertell, Alekh Agarwal +2
In this paper, we prove that Distributional Reinforcement Learning (DistRL), which learns the return distribution, can obtain second-order bounds in both online and offline RL in g…
Multi-Armed Bandits with Interference
Su Jia, Peter Frazier, Nathan Kallus
Experimentation with interference poses a significant challenge in contemporary online platforms. Prior research on experimentation with interference has concentrated on the final…
Low-Rank MDPs with Continuous Action Spaces
Andrew Bennett, Nathan Kallus, Miruna Oprescu
Low-Rank Markov Decision Processes (MDPs) have recently emerged as a promising framework within the domain of reinforcement learning (RL), as they allow for provably approximately…
JoinGym: An Efficient Query Optimization Environment for Reinforcement Learning
Kaiwen Wang, Junxiong Wang, Yueying Li +3
Join order selection (JOS) is the problem of ordering join operations to minimize total query execution cost and it is the core NP-hard combinatorial optimization problem of query…
The Benefits of Being Distributional: Small-Loss Bounds for Reinforcement Learning
Kaiwen Wang, Kevin Zhou, Runzhe Wu +2
While distributional reinforcement learning (DistRL) has been empirically effective, the question of when and why it is better than vanilla, non-distributional RL has remained unan…
Provable Offline Preference-Based Reinforcement Learning
Wenhao Zhan, Masatoshi Uehara, Nathan Kallus +2
In this paper, we investigate the problem of offline Preference-based Reinforcement Learning (PbRL) with human feedback where feedback is available in the form of preference betwee…
B-Learner: Quasi-Oracle Bounds on Heterogeneous Causal Effects Under Hidden Confounding
Miruna Oprescu, Jacob Dorn, Marah Ghoummaid +3
Estimating heterogeneous treatment effects from observational data is a crucial task across many fields, helping policy and decision-makers take better actions. There has been rece…
Near-Minimax-Optimal Risk-Sensitive Reinforcement Learning with CVaR
Kaiwen Wang, Nathan Kallus, Wen Sun
In this paper, we study risk-sensitive Reinforcement Learning (RL), focusing on the objective of Conditional Value at Risk (CVaR) with risk tolerance . Starting with multi-arm b…
Offline Minimax Soft-Q-learning Under Realizability and Partial Coverage
Masatoshi Uehara, Nathan Kallus, Jason D. Lee +1
In offline reinforcement learning (RL) we have no opportunity to explore so we must make assumptions that the data is sufficient to guide picking a good policy, taking the form of…
Smooth Non-Stationary Bandits
Su Jia, Qian Xie, Nathan Kallus +1
In many applications of online decision making, the environment is non-stationary and it is therefore crucial to use bandit algorithms that handle changes. Most existing approaches…
Provable Safe Reinforcement Learning with Binary Feedback
Andrew Bennett, Dipendra Misra, Nathan Kallus
Safety is a crucial necessity in many applications of reinforcement learning (RL), whether robotic, automotive, or medical. Many existing approaches to safe RL rely on receiving nu…
Learning Bellman Complete Representations for Offline Policy Evaluation
Jonathan D. Chang, Kaiwen Wang, Nathan Kallus +1
We study representation learning for Offline Reinforcement Learning (RL), focusing on the important task of Offline Policy Evaluation (OPE). Recent work shows that, in contrast to…
Future-Dependent Value-Based Off-Policy Evaluation in POMDPs
Masatoshi Uehara, Haruka Kiyohara, Andrew Bennett +5
We study off-policy evaluation (OPE) for partially observable MDPs (POMDPs) with general function approximation. Existing methods such as sequential importance sampling estimators…