106 citations · 636 across the 106 of their papers we have counts for
30 papers · 1 filter
The Partial Testimony of Logs: Evaluation of Language Model Generation under Confounded Model Choice
Jikai Jin, Vasilis Syrgkanis
Offline evaluation of language models from usage logs is biased when model choice is confounded: the same user-side factors that influence which model is used can also influence ho…
Prescriptive Scaling Reveals the Evolution of Language Model Capabilities
Hanlin Zhang, Jikai Jin, Vasilis Syrgkanis +1
Machine learning model performance improvements tend to arise from competition and application. For deployment, we consider prescriptive scaling laws: given a pre-training compute…
Policy Learning with Abstention
Ayush Sawarni, Jikai Jin, Justin Whitehouse +1
Policy learning algorithms are widely used in areas such as personalized medicine and advertising to develop individualized treatment regimes. However, most methods force a decisio…
Learning Treatment Representations for Downstream Instrumental Variable Regression
Shiangyi Lin, Hui Lan, Vasilis Syrgkanis
Traditional instrumental variable (IV) estimators face a fundamental constraint: they can only accommodate as many endogenous treatment variables as available instruments. This lim…
Discovering Hierarchical Latent Capabilities of Language Models via Causal Representation Learning
Jikai Jin, Vasilis Syrgkanis, Sham Kakade +1
Faithful evaluation of language model capabilities is crucial for deriving actionable insights that can inform model development. However, rigorous causal evaluations in this domai…
Preference Learning with Response Time: Robust Losses and Guarantees
Ayush Sawarni, Sahasrajit Sarmasarkar, Vasilis Syrgkanis
This paper investigates the integration of response time data into human preference learning frameworks for more effective reward model elicitation. While binary preference data ha…