8 papers
ShellElastic: Scalable Elastic Pretraining with Simultaneous Maximal Updates and Hyperparameter Transfer
Can Jin, Yan Kang, Yifan Gong +5
Foundation models need to support diverse inference budgets, yet training a separate model for each budget duplicates pretraining effort. Elastic models train several nested sub-ne…
Unlearning Is Not Just Erasing: Temporal Decoupling via Generation Inequality
Xunlei Chen, Qirui Ye, Yuang Li +5
Large language models (LLMs) require effective unlearning to address privacy regulations and safety concerns. However, achieving precise forgetting without compromising general uti…
HiLo-Token: Input-Adaptive High-Low Frequency Token Compression for Efficient Image Editing
Haoran You, Yotam Nitzan, Lingzhi Zhang +8
Creative image editing tools, such as Adobe Photoshop's Remove and Generative Fill buttons, are central to everyday customer use and account for a major share of traffic. However,…
Complete-muE: Optimal Hyperparameter Transfer and Scaling for MoE Models
Hongwu Peng, Ohiremen Dibua, Yuanjun Xiong +8
We propose Complete-muE, a framework which targets hyperparameter transfer across dense FFN and any Mixture-of-Experts (MoE) setups in transformer blocks. Existing tools such as $μ…
ALTER: Asymmetric LoRA for Token-Entropy-Guided Unlearning of LLMs
Xunlei Chen, Jinyu Guo, Yuang Li +5
Large language models (LLMs) have advanced to encompass extensive knowledge across diverse domains. Yet controlling what a LLMs should not know is important for ensuring alignment…
DTop-p MoE: Sparsity-Controlled Dynamic Top-p MoE for Foundation Model Pre-training
Can Jin, Hongwu Peng, Mingcan Xiang +7
Sparse Mixture-of-Experts architectures are essential for scaling model capacity efficiently, yet the standard Top- routing imposes a rigid sparsity pattern that ignores the int…
RECALL: REpresentation-aligned Catastrophic-forgetting ALLeviation via Hierarchical Model Merging
Bowen Wang, Haiyuan Wan, Liwen Shi +10
We unveil that internal representations in large language models (LLMs) serve as reliable proxies of learned knowledge, and propose RECALL, a novel representation-aware model mergi…
Hybrid Attention-based Encoder-decoder Model for Efficient Language Model Adaptation
Shaoshi Ling, Guoli Ye, Rui Zhao +1
The attention-based encoder-decoder (AED) speech recognition model has been widely successful in recent years. However, the joint optimization of acoustic model and language model…