10 citations · 10 across the 6 of their papers we have counts for
6 papers
Multi-Head Attention as a Source of Catastrophic Forgetting in MoE Transformers
Anrui Chen, Ruijun Huang, Xin Zhang +15
Mixture-of-Experts (MoE) architectures are often considered a natural fit for continual learning because sparse routing should localize updates and reduce interference, yet MoE Tra…
SD-MoE: Spectral Decomposition for Effective Expert Specialization
Ruijun Huang, Fang Dong, Xin Zhang +16
Mixture-of-Experts (MoE) architectures scale Large Language Models via expert specialization induced by conditional computation. In practice, however, expert specialization often f…
Spectra: Rethinking Optimizers for LLMs Under Spectral Anisotropy
Zhendong Huang, Hengjie Cao, Fang Dong +14
Gradient signals in LLM training are highly anisotropic: recurrent linguistic structure concentrates energy into a small set of dominant spectral directions, while context specific…
Denoising Reuse: Exploiting Inter-frame Motion Consistency for Efficient Video Latent Generation
Chenyu Wang, Shuo Yan, Yixuan Chen +10
Video generation using diffusion-based models is constrained by high computational costs due to the frame-wise iterative diffusion process. This work presents a Diffusion Reuse MOt…
Train Faster, Perform Better: Modular Adaptive Training in Over-Parameterized Models
Yubin Shi, Yixuan Chen, Mingzhi Dong +10
Despite their prevalence in deep-learning communities, over-parameterized models convey high demands of computational costs for proper training. This work studies the fine-grained,…
Can Large Language Models Be Good Companions? An LLM-Based Eyewear System with Conversational Common Ground
Zhenyu Xu, Hailin Xu, Zhouyang Lu +11
Developing chatbots as personal companions has long been a goal of artificial intelligence researchers. Recent advances in Large Language Models (LLMs) have delivered a practical s…