16 papers
The Sparsity Whisperer
Linghao Kong, Inimai Subramanian, Micah Adler +3
Pruning reduces the inference cost of large language models, but existing criteria primarily preserve large activations or reconstruct layer outputs. We argue that this overlooks a…
Expand Neurons, Not Parameters
Linghao Kong, Inimai Subramanian, Yonadav Shavit +3
This work demonstrates how increasing the number of neurons in a network without increasing its total number of non-zero parameters improves performance. We show that this gain cor…
Toy Combinatorial Interpretability Models Reveal Lottery Tickets in Early Feature Space
Alon Bebchuk, Nir Shavit
The lottery ticket hypothesis posits that dense networks contain sparse subnetworks, ``winning tickets,'' that, when rewound to their initial weights and retrained in isolation, ma…
An Interpretable Latency Model for Speculative Decoding in LLM Serving
Linghao Kong, Megan Flynn, Michael Peng +3
Speculative decoding (SD) accelerates large language model (LLM) inference by using a smaller draft model to propose multiple tokens that are verified by a larger target model in p…
Learning to Interpret Weight Differences in Language Models
Avichal Goel, Yoon Kim, Nir Shavit +1
Finetuning (pretrained) language models is a standard approach for updating their internal parametric knowledge and specializing them to new tasks and domains. However, the corresp…
Scalable Energy-Based Models via Adversarial Training: Unifying Discrimination and Generation
Xuwang Yin, Claire Zhang, Julie Steele +2
Simultaneously achieving robust classification and high-fidelity generative modeling within a single framework presents a significant challenge. Hybrid approaches, such as Joint En…