7 papers · 1 filter
LongStraw: Long-Context RL Beyond 2M Tokens under a Fixed GPU Budget
Changhai Zhou, Kieran Liu, Yuhua Zhou +17
Long-context RL post-training is constrained by the lifetime of state and gradients, not attention cost alone. In GRPO, one multi-million-token prompt must serve old-policy and ref…
Holistic Scaling Laws for Optimal Mixture-of-Experts Architecture Optimization
Weilin Wan, Jingtao Han, Weizhong Zhang +2
Scaling laws govern macroscopic resource allocation for LLMs, yet precise architectural configurations for Mixture-of-Experts (MoE) models remain guided by heuristics. Existing MoE…
AutoQRA: Joint Optimization of Mixed-Precision Quantization and Low-rank Adapters for Efficient LLM Fine-Tuning
Changhai Zhou, Shiyang Zhang, Yuhua Zhou +5
Quantization followed by parameter-efficient fine-tuning has emerged as a promising paradigm for downstream adaptation under tight GPU memory constraints. However, this sequential…
Explore and Establish Synergistic Effects Between Weight Pruning and Coreset Selection in Neural Network Training
Weilin Wan, Fan Yi, Weizhong Zhang +2
Modern deep neural networks rely heavily on massive model weights and training samples, incurring substantial computational costs. Weight pruning and coreset selection are two emer…
Computational Budget Should Be Considered in Data Selection
Weilin Wan, Weizhong Zhang, Cheng Jin
Data selection improves computational efficiency by choosing informative subsets of training samples. However, existing methods ignore the compute budget, treating data selection a…
Balancing Fidelity and Plasticity: Aligning Mixed-Precision Fine-Tuning with Linguistic Hierarchies
Changhai Zhou, Shiyang Zhang, Yuhua Zhou +5
Deploying and fine-tuning Large Language Models (LLMs) on resource-constrained edge devices requires navigating a strict trade-off between memory footprint and task performance. Wh…
Large Language Model Compression with Global Rank and Sparsity Optimization
Changhai Zhou, Qian Qiao, Yuhua Zhou +4
Low-rank and sparse composite approximation is a natural idea to compress Large Language Models (LLMs). However, such an idea faces two primary challenges that adversely affect the…