11 citations · 12 across the 10 of their papers we have counts for
12 papers
Inference Auctions
Keegan Harris, Siddharth Prasad, Asher Trockman +2
When inference demand exceeds available compute capacity, model providers must decide which requests should be served first. Users have different tolerances for delay from an LLM A…
Multi-Mixer Models: Flexible Sequence Modeling with Shared Representations
Kevin Y. Li, Asher Trockman, Ananda Theertha Suresh +1
Softmax attention is the cornerstone of modern large language models, but its memory scales linearly and compute quadratically with sequence length. Linear recurrent models, such a…
In-Context Credit Assignment via the Core
Keegan Harris, Siddharth Prasad, Asher Trockman
We propose incentive-aligned mechanisms for in-context credit assignment: the task of assigning credit for AI-generated content among creators whose intellectual property appears i…
Mimetic Initialization of MLPs
Asher Trockman, J. Zico Kolter
Mimetic initialization uses pretrained models as case studies of good initialization, using observations of structures in trained weights to inspire new, simple initialization tech…
Antidistillation Fingerprinting
Yixuan Even Xu, John Kirchenbauer, Yash Savani +5
Model distillation enables efficient emulation of frontier large language models (LLMs), creating a need for robust mechanisms to detect when a third-party student model has traine…
CafeQ: Calibration-free Quantization via Learned Transformations and Adaptive Rounding
Ziteng Sun, Adrian Benton, Samuel Kushnir +4
Post-training quantization is an effective method for reducing the serving cost of large language models, where the standard approach is to use a round-to-nearest quantization leve…
Antidistillation Sampling
Yash Savani, Asher Trockman, Zhili Feng +5
Frontier models that generate extended reasoning traces inadvertently produce rich token sequences that can facilitate model distillation. Recognizing this vulnerability, model own…
Mimetic Initialization Helps State Space Models Learn to Recall
Asher Trockman, Hrayr Harutyunyan, J. Zico Kolter +2
Recent work has shown that state space models such as Mamba are significantly worse than Transformers on recall-based tasks due to the fact that their state size is constant with r…
Mimetic Initialization of Self-Attention Layers
Asher Trockman, J. Zico Kolter
It is notoriously difficult to train Transformers on small datasets; typically, large pre-trained models are instead used as the starting point. We explore the weights of such pre-…
Understanding the Covariance Structure of Convolutional Filters
Asher Trockman, Devin Willmott, J. Zico Kolter
Neural network weights are typically initialized at random from univariate distributions, controlling just the variance of individual weights even in highly-structured operations l…
Patches Are All You Need?
Asher Trockman, J. Zico Kolter
Although convolutional networks have been the dominant architecture for vision tasks for many years, recent experiments have shown that Transformer-based models, most notably the V…
Orthogonalizing Convolutional Layers with the Cayley Transform
Asher Trockman, J. Zico Kolter
Recent work has highlighted several advantages of enforcing orthogonality in the weight layers of deep networks, such as maintaining the stability of activations, preserving gradie…