71 citations · 186 across the 38 of their papers we have counts for
40 papers
LLM Evaluation is a Forecasting Problem
Rahul Thomas, Adam Block, Micah Goldblum
Standard procedure in machine learning is to randomly split train and test data, emulating IID draws from one underlying data distribution. In practice, models are trained on past…
Toward Simulations of Cold Baryogenesis via Supercooled Phase Transitions
Naoya Kitajima, Shota Nakagawa, Yuichiro Nakai +2
Supercooled first-order phase transitions provide an attractive framework for generating observable gravitational waves (GWs), but the accompanying entropy injection dilutes any pr…
Pinocchio: Fast Uncertainty Estimates for Black-Box Language Models
Kevin David Hayes, Arka Pal, Haosong Zhang +2
In high-stakes decision-making applications of large language models (LLMs), practitioners require not only accurate LLMs but also uncertainty estimates for their predictions. Exis…
Recovering Wasted Compute in Autoresearch Agents
Au Kwok Chun, Abhigyan Acherjee, Amrutha Rao +4
A slew of recent works develop agents for solving research problems end-to-end, a paradigm increasingly referred to as autoresearch. Such agents have inspired large industry invest…
Understanding Reasoning from Pretraining to Post-Training
Jingyan Shen, Ang Li, Salman Rahman +4
Reinforcement learning (RL) has become central to improving large language models (LLMs) on complex reasoning tasks, yet RL post-training is largely studied in isolation from the p…
End-to-End Context Compression at Scale
Ang Li, Sean McLeish, Haozhe Chen +12
Long-context language model inference is bottlenecked by memory, as the KV cache grows with context length. Recent techniques to compress the KV cache fall short: they either degra…
On Training in Imagination
Nadav Timor, Ravid Shwartz-Ziv, Micah Goldblum +2
State-of-the-art model-based reinforcement learning methods train policies on imagined rollouts. These rollouts are trajectories generated by a learned dynamics model and are score…
LiveMathematicianBench: A Live Benchmark for Research-Level Mathematical Reasoning with Proof Sketches
Linyang He, Qiyao Yu, Hanze Dong +5
Mathematical reasoning is a hallmark of human intelligence, and whether large language models (LLMs) can meaningfully perform it remains a central question in artificial intelligen…
Privacy-Preserving Mechanisms Enable Cheap Verifiable Inference of LLMs
Arka Pal, Louai Zahran, William Gvozdjak +2
As large language models (LLMs) continue to grow in size, fewer users are able to host and run models locally. This has led to increased use of third-party hosting services. Howeve…
Dynamic Delayed Tree Expansion For Improved Multi-Path Speculative Decoding
Rahul Thomas, Teo Kitanovski, Micah Goldblum +1
Multi-path speculative decoding accelerates lossless sampling from a target model by using a cheaper draft model to generate a draft tree of tokens, and then applies a verification…
Closing the Train-Test Gap in World Models for Gradient-Based Planning
Arjun Parthasarathy, Nimit Kalra, Rohun Agrawal +4
World models paired with model predictive control (MPC) can be trained offline on large-scale datasets of expert trajectories and enable generalization to a wide range of planning…
Preserving Cross-Modal Consistency for CLIP-based Class-Incremental Learning
Haoran Chen, Houze Xu, Micah Goldblum +2
Class-incremental learning (CIL) enables models to continuously learn new categories from sequential tasks without forgetting previously acquired knowledge. While recent advances i…
Knowing What You Know Is Not Enough: Large Language Model Confidences Don't Align With Their Actions
Arka Pal, Teo Kitanovski, Arthur Liang +2
Large language models (LLMs) are increasingly deployed in agentic and multi-turn workflows where they are tasked to perform actions of significant consequence. In order to deploy t…
An Attack to Break Permutation-Based Private Third-Party Inference Schemes for LLMs
Rahul Thomas, Louai Zahran, Erica Choi +3
Recent advances in Large Language Models (LLMs) have led to the widespread adoption of third-party inference services, raising critical privacy concerns. Existing methods of perfor…
vTune: Verifiable Fine-Tuning for LLMs Through Backdooring
Eva Zhang, Arka Pal, Akilesh Potti +1
As fine-tuning large language models (LLMs) becomes increasingly prevalent, users often rely on third-party services with limited visibility into their fine-tuning processes. This…
Searching for Efficient Linear Layers over a Continuous Space of Structured Matrices
Andres Potapczynski, Shikai Qiu, Marc Finzi +6
Dense linear layers are the dominant computational bottleneck in large neural networks, presenting a critical need for more efficient alternatives. Previous efforts focused on a sm…
Style Outweighs Substance: Failure Modes of LLM Judges in Alignment Benchmarking
Benjamin Feuer, Micah Goldblum, Teresa Datta +5
The release of ChatGPT in November 2022 sparked an explosion of interest in post-training and an avalanche of new preference optimization (PO) methods. These methods claim superior…
Unlocking Tokens as Data Points for Generalization Bounds on Larger Language Models
Sanae Lotfi, Yilun Kuang, Brandon Amos +3
Large language models (LLMs) with billions of parameters excel at predicting the next token in a sequence. Recent work computes non-vacuous compression-based generalization bounds…
Compute Better Spent: Replacing Dense Layers with Structured Matrices
Shikai Qiu, Andres Potapczynski, Marc Finzi +2
Dense linear layers are the dominant computational bottleneck in foundation models. Identifying more efficient alternatives to dense matrices has enormous potential for building mo…
Large Language Models Must Be Taught to Know What They Don't Know
Sanyam Kapoor, Nate Gruver, Manley Roberts +7
When using large language models (LLMs) in high-stakes applications, we need to know when we can trust their predictions. Some works argue that prompting high-performance LLMs is s…
Adaptive Retention & Correction: Test-Time Training for Continual Learning
Haoran Chen, Micah Goldblum, Zuxuan Wu +1
Continual learning, also known as lifelong learning or incremental learning, refers to the process by which a model learns from a stream of incoming data over time. A common proble…
TuneTables: Context Optimization for Scalable Prior-Data Fitted Networks
Benjamin Feuer, Robin Tibor Schirrmeister, Valeriia Cherepanova +5
While tabular classification has traditionally relied on from-scratch training, a recent breakthrough called prior-data fitted networks (PFNs) challenges this approach. Similar to…
Simplifying Neural Network Training Under Class Imbalance
Ravid Shwartz-Ziv, Micah Goldblum, Yucen Lily Li +2
Real-world datasets are often highly class-imbalanced, which can adversely impact the performance of deep learning models. The majority of research on training neural networks unde…
Non-Vacuous Generalization Bounds for Large Language Models
Sanae Lotfi, Marc Finzi, Yilun Kuang +3
Modern language models can contain billions of parameters, raising the question of whether they can generalize beyond the training data or simply parrot their training corpora. We…