9 papers
How Data Shapes RoPE Frequency Usage: From Positional Scale Matching to Length Generalization
Xinyi Wu, Siyuan Liu, Ali Jadbabaie
Rotary Position Embeddings (RoPE) provide transformers with a fixed grid of positional frequencies, yet trained models use these frequencies highly non-uniformly. We study what det…
Unbiased Regression-Adjusted Estimation of Average Treatment Effects in Randomized Controlled Trials
Alberto Abadie, Mehrdad Ghadiri, Ali Jadbabaie +1
This article introduces a leave-one-out regression adjustment (LOORA) for estimating average treatment effects in randomized controlled trials. In finite samples, LOORA removes the…
Is Your Conditional Diffusion Model Actually Denoising?
Daniel Pfrommer, Zehao Dou, Christopher Scarvelis +2
We study the inductive biases of diffusion models with a conditioning-variable, which have seen widespread application as both text-conditioned generative image models and observat…
A Test-Function Approach to Incremental Stability
Daniel Pfrommer, Max Simchowitz, Ali Jadbabaie
This paper presents a novel framework for analyzing Incremental-Input-to-State Stability (ISS) based on the idea of using rewards as "test functions." Whereas control theory tr…
Fast Tensor Completion via Approximate Richardson Iteration
Mehrdad Ghadiri, Matthew Fahrbach, Yunbum Kook +1
We study tensor completion (TC) through the lens of low-rank tensor decomposition (TD). Many TD algorithms use fast alternating minimization methods to solve highly structured line…
On the Emergence of Position Bias in Transformers
Xinyi Wu, Yifei Wang, Stefanie Jegelka +1
Recent studies have revealed various manifestations of position bias in transformer architectures, from the "lost-in-the-middle" phenomenon to attention sinks, yet a comprehensive…