6 papers
The Partial Testimony of Logs: Evaluation of Language Model Generation under Confounded Model Choice
Jikai Jin, Vasilis Syrgkanis
Offline evaluation of language models from usage logs is biased when model choice is confounded: the same user-side factors that influence which model is used can also influence ho…
Prescriptive Scaling Reveals the Evolution of Language Model Capabilities
Hanlin Zhang, Jikai Jin, Vasilis Syrgkanis +1
Machine learning model performance improvements tend to arise from competition and application. For deployment, we consider prescriptive scaling laws: given a pre-training compute…
Adaptive Exploration for Latent-State Bandits
Jikai Jin, Kenneth Hung, Sanath Kumar Krishnamurthy +2
We study bandits whose rewards depend on an unobserved Markov state that evolves independently of the learner's actions. The optimal arm can change even though the learner observes…
Sharp Structure-Agnostic Lower Bounds for General Linear Functional Estimation
Jikai Jin, Vasilis Syrgkanis
We establish a general statistical optimality theory for estimation problems where the target parameter is a linear functional of an unknown nuisance component that must be estimat…
Policy Learning with Abstention
Ayush Sawarni, Jikai Jin, Justin Whitehouse +1
Policy learning algorithms are widely used in areas such as personalized medicine and advertising to develop individualized treatment regimes. However, most methods force a decisio…
It's Hard to Be Normal: The Impact of Noise on Structure-agnostic Estimation
Jikai Jin, Lester Mackey, Vasilis Syrgkanis
Structure-agnostic causal inference studies how well one can estimate a treatment effect given black-box machine learning estimates of nuisance functions (like the impact of confou…