16 papers
Stepped MoE: Segment-Level Routing with Configurable Inference Complexity
Arnav Kundu, Zhaoyang Xu, Bairu Hou +3
Training large language models (LLMs) is resource-intensive, and adapting them for diverse deployment scenarios with varying computational constraints remains challenging. While el…
Efficient Mixture-of-Experts with Speculative Decoding via Expert Coactivation
Kumari Nishu, Han-Byul Kim, Santosh Chilkunda +13
Mixture-of-Experts (MoE) models are increasingly deployed alongside Speculative Decoding (SD) to accelerate inference, but combining the two is challenging. SD improves the inferen…
MemoryLLM: Plug-n-Play Interpretable Feed-Forward Memory for Transformers
Ajay Jaiswal, Lauren Hannah, Han-Byul Kim +4
Understanding how transformer components operate in LLMs is important, as it is at the core of recent technological advances in artificial intelligence. In this work, we revisit th…
MoEs Are Stronger than You Think: Hyper-Parallel Inference Scaling with RoE
Soheil Zibakhsh, Mohammad Samragh, Kumari Nishu +3
The generation quality of large language models (LLMs) is often improved by utilizing inference-time sequence-level scaling methods (e.g., Chain-of-Thought). We introduce hyper-par…
Mirror Speculative Decoding: Breaking the Serial Barrier in LLM Inference
Nikhil Bhendawade, Kumari Nishu, Arnav Kundu +3
Speculative decoding accelerates LLM inference by using a draft model to look ahead, but gains are capped by the cost of autoregressive draft generation: increasing draft size elev…
MoE-PHDS: One MoE checkpoint for flexible runtime sparsity
Lauren. A Hannah, Soheil Zibakhsh, Kumari Nishu +4
Sparse Mixtures of Experts (MoEs) are typically trained to operate at a fixed sparsity level, e.g. in a top- gating function. This global sparsity level determines an operat…
EpiCache: Episodic KV Cache Management for Long-Term Conversation on Resource-Constrained Environments
Minsoo Kim, Arnav Kundu, Han-Byul Kim +2
Modern large language models (LLMs) extend context lengths to millions of tokens, enabling coherent, personalized responses grounded in long conversational history. However, the Ke…
Your LLM Knows the Future: Uncovering Its Multi-Token Prediction Potential
Mohammad Samragh, Arnav Kundu, David Harrison +4
Autoregressive language models are constrained by their inherently sequential nature, generating one token at a time. This paradigm limits inference speed and parallelism, especial…
SPD: Sync-Point Drop for Efficient Tensor Parallelism of Large Language Models
Han-Byul Kim, Duc Hoang, Arnav Kundu +2
With the rapid expansion in the scale of large language models (LLMs), enabling efficient distributed inference across multiple computing units has become increasingly critical. Ho…
An Efficient and Streaming Audio Visual Active Speaker Detection System
Arnav Kundu, Yanzi Jin, Mohammad Sekhavat +3
This paper delves into the challenging task of Active Speaker Detection (ASD), where the system needs to determine in real-time whether a person is speaking or not in a series of v…
RepCNN: Micro-sized, Mighty Models for Wakeword Detection
Arnav Kundu, Prateeth Nayak, Priyanka Padmanabhan +1
Always-on machine learning models require a very low memory and compute footprint. Their restricted parameter count limits the model's capacity to learn, and the effectiveness of t…
Streaming Anchor Loss: Augmenting Supervision with Temporal Significance
Utkarsh Oggy Sarawgi, John Berkowitz, Vineet Garg +5
Streaming neural network models for fast frame-wise responses to various speech and sensory signals are widely adopted on resource-constrained platforms. Hence, increasing the lear…
R2 Loss: Range Restriction Loss for Model Compression and Quantization
Arnav Kundu, Chungkuk Yoo, Srijan Mishra +2
Model quantization and compression is widely used techniques to reduce usage of computing resource at inference time. While state-of-the-art works have been achieved reasonable acc…
HEiMDaL: Highly Efficient Method for Detection and Localization of wake-words
Arnav Kundu, Mohammad Samragh Razlighi, Minsik Cho +2
Streaming keyword spotting is a widely used solution for activating voice assistants. Deep Neural Networks with Hidden Markov Model (DNN-HMM) based methods have proven to be effici…
I see what you hear: a vision-inspired method to localize words
Mohammad Samragh, Arnav Kundu, Ting-Yao Hu +5
This paper explores the possibility of using visual object detection techniques for word localization in speech data. Object detection has been thoroughly studied in the contempora…
Optimize what matters: Training DNN-HMM Keyword Spotting Model Using End Metric
Ashish Shrivastava, Arnav Kundu, Chandra Dhir +2
Deep Neural Network--Hidden Markov Model (DNN-HMM) based methods have been successfully used for many always-on keyword spotting algorithms that detect a wake word to trigger a dev…