12 citations · 18 across the 9 of their papers we have counts for
9 papers · 1 filter
Adaptive-Depth Is a Finishing Stage
Andrei Piterbarg, Sean McLeish, Dhruv Pai +1
Looped transformers reuse the same layer multiple times, allowing inference compute to be adjusted without changing the model. Recent adaptive-depth methods train the model across…
Aurora: A Leverage-Aware Spectral Optimizer
Alec Dewulf, Dhruv Pai, Li Yang +2
We show that for tall matrix parameters, like projection matrices in the MLP layers, the Muon update can have row norms that are arbitrarily non-uniform. This can lead to a self-re…
Parallax: Parameterized Local Linear Attention for Language Modeling
Yifei Zuo, Dhruv Pai, Zhichen Zeng +3
Large Language Models (LLMs) have become the central paradigm in artificial intelligence, yet the core computational primitive of attention has remained structurally unchanged. Loc…
PreFT: Prefill-only finetuning for efficient inference
Andrew Lanpouthakoun, Aryaman Arora, Zhengxuan Wu +4
Large language models can now be personalised efficiently at scale using parameter efficient finetuning methods (PEFTs), but serving user-specific PEFTs harms throughput, even with…
Towards an Improved Understanding and Utilization of Maximum Manifold Capacity Representations
Rylan Schaeffer, Victor Lecomte, Dhruv Bhandarkar Pai +10
Maximum Manifold Capacity Representations (MMCR) is a recent multi-view self-supervised learning (MVSSL) method that matches or surpasses other leading MVSSL methods. MMCR is intri…
Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data
Matthias Gerstgrasser, Rylan Schaeffer, Apratim Dey +11
The proliferation of generative models, combined with pretraining on web-scale data, raises a timely question: what happens when these models are trained on their own generated out…
Bridging Associative Memory and Probabilistic Modeling
Rylan Schaeffer, Nika Zahedi, Mikail Khona +9
Associative memory and probabilistic modeling are two fundamental topics in artificial intelligence. The first studies recurrent neural networks designed to denoise, complete and r…
Deceptive Alignment Monitoring
Andres Carranza, Dhruv Pai, Rylan Schaeffer +2
As the capabilities of large machine learning models continue to grow, and as the autonomy afforded to such models continues to expand, the spectre of a new adversary looms: the mo…
FACADE: A Framework for Adversarial Circuit Anomaly Detection and Evaluation
Dhruv Pai, Andres Carranza, Rylan Schaeffer +2
We present FACADE, a novel probabilistic and geometric framework designed for unsupervised mechanistic anomaly detection in deep neural networks. Its primary goal is advancing the…