5 papers
GS-MoE: Training-Free MoE Compression via Groupwise Subspace Sharing and Pruning
Carson Cary, Yukai Jin, Yihao Huang +5
Mixture-of-Experts (MoE) models reduce computation through sparse activation, but storing their expert parameters remains a major memory bottleneck. We introduce GS-MoE, a training…
Learning from Synthetic Data without Model Collapse in Iterative Instruction Tuning
Xiaonan Luo, Yue Huang, Kehan Guo +4
Model collapse is a central challenge in learning from synthetic data: as later-generation large language models (LLMs) are trained on an increasing proportion of model-generated d…
HyperEdit: Unlocking Instruction-based Text Editing in LLMs via Hypernetworks
Yiming Zeng, Jinghan Cao, Zexin Li +7
Instruction-based text editing is increasingly critical for real-world applications such as code editors (e.g., Cursor), but Large Language Models (LLMs) continue to struggle with…
Dynamic Noise Preference Optimization: Self-Improvement of Large Language Models with Self-Synthetic Data
Haoyan Yang, Khiem Le, Ting Hua +7
Although LLMs have achieved significant success, their reliance on large volumes of human-annotated data has limited their potential for further scaling. In this situation, utilizi…
ToMoE: Converting Dense Large Language Models to Mixture-of-Experts through Dynamic Structural Pruning
Shangqian Gao, Ting Hua, Reza Shirkavand +10
Large Language Models (LLMs) have demonstrated remarkable abilities in tackling a wide range of complex tasks. However, their huge computational and memory costs raise significant…