Publications (15)
Kernel Approximation Methods for Speech Recognition
Avner May, Alireza Bagheri Garakani, Zhiyun Lu +9
We study large-scale kernel methods for acoustic modeling in speech recognition and compare their performance to deep neural networks (DNNs). We perform experiments on four speech…
Audio-visual fine-tuning of audio-only ASR models
Avner May, Dmitriy Serdyuk, Ankit Parag Shah +2
Audio-visual automatic speech recognition (AV-ASR) models are very effective at reducing word error rates on noisy speech, but require large amounts of transcribed AV training data…
Low-Precision Random Fourier Features for Memory-Constrained Kernel Approximation
Jian Zhang, Avner May, Tri Dao +1
We investigate how to train kernel approximation methods that generalize well under a memory budget. Building on recent theoretical work, we define a measure of kernel approximatio…
MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decoding
Ranajoy Sadhukhan, Jian Chen, Zhuoming Chen +7
Large Language Models (LLMs) have become more prevalent in long-context applications such as interactive chatbots, document analysis, and agent workflows, but it is challenging to…
A Comparison between Deep Neural Nets and Kernel Acoustic Models for Speech Recognition
Zhiyun Lu, Dong Guo, Alireza Bagheri Garakani +8
We study large-scale kernel methods for acoustic modeling and compare to DNNs on performance metrics related to both acoustic modeling and recognition. Measuring perplexity and fra…
Minions: Cost-efficient Collaboration Between On-device and Cloud Language Models
Avanika Narayan, Dan Biderman, Sabri Eyuboglu +4
We investigate an emerging setup in which a small, on-device language model (LM) with access to local data communicates with a frontier, cloud-hosted LM to solve real-world tasks i…
Understanding the Downstream Instability of Word Embeddings
Megan Leszczynski, Avner May, Jian Zhang +3
Many industrial machine learning (ML) systems require frequent retraining to keep up-to-date with constantly changing data. This retraining exacerbates a large challenge facing ML…
Speculative Speculative Decoding
Tanishq Kumar, Tri Dao, Avner May
Autoregressive decoding is bottlenecked by its sequential nature. Speculative decoding has become a standard way to accelerate inference by using a fast draft model to predict upco…
The Mamba in the Llama: Distilling and Accelerating Hybrid Models
Junxiong Wang, Daniele Paliotta, Avner May +2
Linear RNN architectures, like Mamba, can be competitive with Transformer models in language modeling while having advantageous deployment characteristics. Given the focus on train…
When RL Meets Adaptive Speculative Training: A Unified Training-Serving System
Junxiong Wang, Fengxiang Bie, Jisen Li +14
Speculative decoding can significantly accelerate LLM serving, yet most deployments today disentangle speculator training from serving, treating speculator training as a standalone…
On the Downstream Performance of Compressed Word Embeddings
Avner May, Jian Zhang, Tri Dao +1
Compressing word embeddings is important for deploying NLP models in memory-constrained settings. However, understanding what makes compressed embeddings perform well on downstream…
How to Scale Up Kernel Methods to Be As Good As Deep Neural Nets
Zhiyun Lu, Avner May, Kuan Liu +8
The computational complexity of kernel methods has often been a major barrier for applying them to large-scale learning problems. We argue that this barrier can be effectively over…
SpecExec: Massively Parallel Speculative Decoding for Interactive LLM Inference on Consumer Devices
Ruslan Svirschevski, Avner May, Zhuoming Chen +3
As large language models gain widespread adoption, running them efficiently becomes crucial. Recent works on LLM inference use speculative decoding to achieve extreme speedups. How…
Sequoia: Scalable, Robust, and Hardware-aware Speculative Decoding
Zhuoming Chen, Avner May, Ruslan Svirschevski +4
As the usage of large language models (LLMs) grows, performing efficient inference with these models becomes increasingly important. While speculative decoding has recently emerged…
Contextual Embeddings: When Are They Worth It?
Simran Arora, Avner May, Jian Zhang +1
We study the settings for which deep contextual embeddings (e.g., BERT) give large improvements in performance relative to classic pretrained embeddings (e.g., GloVe), and an even…