5 papers · 1 filter
How Data Shapes RoPE Frequency Usage: From Positional Scale Matching to Length Generalization
Xinyi Wu, Siyuan Liu, Ali Jadbabaie
Rotary Position Embeddings (RoPE) provide transformers with a fixed grid of positional frequencies, yet trained models use these frequencies highly non-uniformly. We study what det…
Is Your Conditional Diffusion Model Actually Denoising?
Daniel Pfrommer, Zehao Dou, Christopher Scarvelis +2
We study the inductive biases of diffusion models with a conditioning-variable, which have seen widespread application as both text-conditioned generative image models and observat…
A Test-Function Approach to Incremental Stability
Daniel Pfrommer, Max Simchowitz, Ali Jadbabaie
This paper presents a novel framework for analyzing Incremental-Input-to-State Stability (ISS) based on the idea of using rewards as "test functions." Whereas control theory tra…
The Pitfalls of Imitation Learning when Actions are Continuous
Max Simchowitz, Daniel Pfrommer, Ali Jadbabaie
We study the problem of imitating an expert demonstrator in a discrete-time, continuous state-and-action control system. We show that, even if the dynamics satisfy a control-theore…
On the Emergence of Position Bias in Transformers
Xinyi Wu, Yifei Wang, Stefanie Jegelka +1
Recent studies have revealed various manifestations of position bias in transformer architectures, from the "lost-in-the-middle" phenomenon to attention sinks, yet a comprehensive…