6 papers · 1 filter
Stepped MoE: Segment-Level Routing with Configurable Inference Complexity
Arnav Kundu, Zhaoyang Xu, Bairu Hou +3
Training large language models (LLMs) is resource-intensive, and adapting them for diverse deployment scenarios with varying computational constraints remains challenging. While el…
Efficient Mixture-of-Experts with Speculative Decoding via Expert Coactivation
Kumari Nishu, Han-Byul Kim, Santosh Chilkunda +13
Mixture-of-Experts (MoE) models are increasingly deployed alongside Speculative Decoding (SD) to accelerate inference, but combining the two is challenging. SD improves the inferen…
MemoryLLM: Plug-n-Play Interpretable Feed-Forward Memory for Transformers
Ajay Jaiswal, Lauren Hannah, Han-Byul Kim +4
Understanding how transformer components operate in LLMs is important, as it is at the core of recent technological advances in artificial intelligence. In this work, we revisit th…
MoE-PHDS: One MoE checkpoint for flexible runtime sparsity
Lauren. A Hannah, Soheil Zibakhsh, Kumari Nishu +4
Sparse Mixtures of Experts (MoEs) are typically trained to operate at a fixed sparsity level, e.g. in a top- gating function. This global sparsity level determines an operat…
Streaming Anchor Loss: Augmenting Supervision with Temporal Significance
Utkarsh Oggy Sarawgi, John Berkowitz, Vineet Garg +5
Streaming neural network models for fast frame-wise responses to various speech and sensory signals are widely adopted on resource-constrained platforms. Hence, increasing the lear…
R2 Loss: Range Restriction Loss for Model Compression and Quantization
Arnav Kundu, Chungkuk Yoo, Srijan Mishra +2
Model quantization and compression is widely used techniques to reduce usage of computing resource at inference time. While state-of-the-art works have been achieved reasonable acc…