3 papers
cs.LG2026
Negligible in Size, Significant in Effect: On Scale Vectors in Large Language Models
Mingze Wang, Shuchen Zhu, Yuxin Fang +3
Normalization layers in modern large language models (LLMs) consist of a deterministic normalization operation and a learnable scale vector. While the normalization operation has b…
cs.LG2026
More Expressive Feedforward Layers: Part I. Token-Adaptive Mixing of Activations
Mingze Wang, Jinbo Wang, Yikuan Xia +2
Feedforward network (FFN) layers account for a large fraction of parameters and nonlinear expressivity in Transformer-based large language models (LLMs). Despite the evolution from…
eess.AS2024
Phoneme-Level Contrastive Learning for User-Defined Keyword Spotting with Flexible Enrollment
Li Kewei, Zhou Hengshun, Shen Kai +2
User-defined keyword spotting (KWS) enhances the user experience by allowing individuals to customize keywords. However, in open-vocabulary scenarios, most existing methods commonl…