Latent Utility Q-Learning for Preference-Adaptive Dynamic Treatment Regimes

Abstract

Optimizing individualized treatment sequences for patients who weigh multiple, competing outcomes differently poses a challenge for dynamic treatment regime (DTR) methods, which typically assume a single univariate outcome. We propose Latent Utility Q-Learning (LUQ-Learning), which estimates DTRs optimizing patient-specific preference-weighted combinations of multivariate outcomes $\mathbf{Y}\in {\mathbb{R}}^d$ across $K$ decision points. A conditional mean factorization decouples preference estimation from outcome regression, enabling flexible, modular learning under imperfectly observed and heterogeneous preferences without requiring explicit outcome ranking by patients. We establish consistency of the estimated value function and derive unified $\epsilon$-optimality guarantees that bound policy value loss in terms of posterior preference uncertainty, yielding interpretable criteria for data-driven policy selection. Simulations calibrated to Sequential Multiple Assignment Randomized Trials (SMARTs) demonstrate that LUQ-Learning outperforms Q-learning with naive outcome aggregation, last-reported satisfaction optimization, and existing preference-based methods.

Joshua P. Zitovsky and Yating Zou contributed equally to this work as co-first authors