9 citations · 18 across the 23 of their papers we have counts for
24 papers
CausalDS: Benchmarking Causal Reasoning in Data-Science Agents
Andrej Leban, Yuekai Sun, Theo Saarinen
Large language models (LLMs) increasingly act as data-science agents, combining abstract reasoning with advanced tool use. Yet the relevant benchmark landscape largely divides into…
From Reasoning Traces to Reusable Modules: Understanding Compositional Generalization in Language Model Reasoning
Lingjing Kong, Xin Liu, Guangyi Chen +9
Post-training pipelines that combine supervised fine-tuning (SFT) with reinforcement learning (RL) have emerged as the key recipe for transforming large language models (LLMs) into…
LeanMarathon: Toward Reliable AI Co-Mathematicians through Long-Horizon Lean Autoformalization
Yuanhe Zhang, Yuekai Sun, Taiji Suzuki +2
Long-horizon autoformalization of research mathematics fails not only at hard lemmas, but at scale: statements drift, dependencies tangle, context decays, and local repairs corrupt…
The Open Source Economic Index of AI Adoption and Capability
Seamus Somerstep, Aritra Guha, Divesh Srivastava +1
We work towards measuring both AI adoption and the capability of AI to perform discrete labor tasks across various occupations. To measure adoption, we develop an open-source econo…
K2-V2: A 360-Open, Reasoning-Enhanced LLM
K2 Team, Zhengzhong Liu, Liping Tang +36
We introduce K2-V2, a 360-open LLM built from scratch as a superior base for reasoning adaptation, in addition to functions such as conversation and knowledge retrieval from genera…
A Latent Variable Framework for Scaling Laws in Large Language Models
Peiyao Cai, Chengyu Cui, Felipe Maia Polo +6
We propose a statistical framework built on latent variable modeling for scaling laws of large language models (LLMs). Our work is motivated by the rapid emergence of numerous new…
Limitations of refinement methods for weak to strong generalization
Seamus Somerstep, Ya'acov Ritov, Mikhail Yurochkin +2
Standard techniques for aligning large language models (LLMs) utilize human-produced data, which could limit the capability of any aligned LLM to human level. Label refinement and…
Learning to Choose or Choosing to Learn: Best-of-N vs. Supervised Fine-Tuning for Bit String Generation
Seamus Somerstep, Vinod Raman, Unique Subedi +1
Using the bit string generation problem as a case study, we theoretically compare two standard methods for adapting large language models to new tasks. The first, referred to as su…
Sloth: scaling laws for LLM skills to predict multi-benchmark performance across families
Felipe Maia Polo, Seamus Somerstep, Leshem Choshen +2
Scaling laws for large language models (LLMs) predict model performance based on parameters like size and training data. However, differences in training configurations and data pr…
Microfoundation Inference for Strategic Prediction
Daniele Bracale, Subha Maity, Felipe Maia Polo +3
Often in prediction tasks, the predictive model itself can influence the distribution of the target variable, a phenomenon termed performative prediction. Generally, this influence…
LiveXiv -- A Multi-Modal Live Benchmark Based on Arxiv Papers Content
Nimrod Shabtay, Felipe Maia Polo, Sivan Doveh +8
The large-scale training of multi-modal models on data scraped from the web has shown outstanding utility in infusing these models with the required world knowledge to perform effe…
Algorithmic Fairness in Performative Policy Learning: Escaping the Impossibility of Group Fairness
Seamus Somerstep, Ya'acov Ritov, Yuekai Sun
In many prediction problems, the predictive model affects the distribution of the prediction target. This phenomenon is known as performativity and is often caused by the behavior…
Efficient multi-prompt evaluation of LLMs
Felipe Maia Polo, Ronald Xu, Lucas Weber +6
Most popular benchmarks for comparing LLMs rely on a limited set of prompt templates, which may not fully capture the LLMs' abilities and can affect the reproducibility of results…
A transfer learning framework for weak-to-strong generalization
Seamus Somerstep, Felipe Maia Polo, Moulinath Banerjee +3
Modern large language model (LLM) alignment techniques rely on human feedback, but it is unclear whether these techniques fundamentally limit the capabilities of aligned LLMs. In p…
Learning the Distribution Map in Reverse Causal Performative Prediction
Daniele Bracale, Subha Maity, Moulinath Banerjee +1
In numerous predictive scenarios, the predictive model affects the sampling distribution; for example, job applicants often meticulously craft their resumes to navigate through a s…
Prompt Exploration with Prompt Regression
Michael Feffer, Ronald Xu, Yuekai Sun +1
In the advent of democratized usage of large language models (LLMs), there is a growing desire to systematize LLM prompt creation and selection processes beyond iterative trial-and…
Learning In Reverse Causal Strategic Environments With Ramifications on Two Sided Markets
Seamus Somerstep, Yuekai Sun, Ya'acov Ritov
Motivated by equilibrium models of labor markets, we develop a formulation of causal strategic classification in which strategic agents can directly manipulate their outcomes. As a…
Aligners: Decoupling LLMs and Alignment
Lilian Ngweta, Mayank Agarwal, Subha Maity +3
Large Language Models (LLMs) need to be aligned with human expectations to ensure their safety and utility in most applications. Alignment is challenging, costly, and needs to be r…
tinyBenchmarks: evaluating LLMs with fewer examples
Felipe Maia Polo, Lucas Weber, Leshem Choshen +3
The versatility of large language models (LLMs) led to the creation of diverse benchmarks that thoroughly test a variety of language models' abilities. These benchmarks consist of…
Weak Supervision Performance Evaluation via Partial Identification
Felipe Maia Polo, Subha Maity, Mikhail Yurochkin +2
Programmatic Weak Supervision (PWS) enables supervised model training without direct access to ground truth labels, utilizing weak labels from heuristics, crowdsourcing, or pre-tra…
An Investigation of Representation and Allocation Harms in Contrastive Learning
Subha Maity, Mayank Agarwal, Mikhail Yurochkin +1
The effect of underrepresentation on the performance of minority groups is known to be a serious problem in supervised learning settings; however, it has been underexplored so far…
Fusing Models with Complementary Expertise
Hongyi Wang, Felipe Maia Polo, Yuekai Sun +3
Training AI models that generalize across tasks and domains has long been among the open problems driving AI research. The emergence of Foundation Models made it easier to obtain e…
Large Language Model Routing with Benchmark Datasets
Tal Shnitzer, Anthony Ou, Mírian Silva +5
There is a rapidly growing number of open-source Large Language Models (LLMs) and benchmark datasets to compare them. While some models dominate these benchmarks, no single model t…
Conditional independence testing under misspecified inductive biases
Felipe Maia Polo, Yuekai Sun, Moulinath Banerjee
Conditional independence (CI) testing is a fundamental and challenging task in modern statistics and machine learning. Many modern methods for CI testing rely on powerful supervise…