5 papers
Holistic Scaling Laws for Optimal Mixture-of-Experts Architecture Optimization
Weilin Wan, Jingtao Han, Weizhong Zhang +2
Scaling laws govern macroscopic resource allocation for LLMs, yet precise architectural configurations for Mixture-of-Experts (MoE) models remain guided by heuristics. Existing MoE…
Advanced Black-Box Tuning of Large Language Models with Limited API Calls
Zhikang Xie, Weilin Wan, Peizhu Gong +2
Black-box tuning is an emerging paradigm for adapting large language models (LLMs) to better achieve desired behaviors, particularly when direct access to model parameters is unava…
Explore and Establish Synergistic Effects Between Weight Pruning and Coreset Selection in Neural Network Training
Weilin Wan, Fan Yi, Weizhong Zhang +2
Modern deep neural networks rely heavily on massive model weights and training samples, incurring substantial computational costs. Weight pruning and coreset selection are two emer…
Computational Budget Should Be Considered in Data Selection
Weilin Wan, Weizhong Zhang, Cheng Jin
Data selection improves computational efficiency by choosing informative subsets of training samples. However, existing methods ignore the compute budget, treating data selection a…
Out-of-Distribution Detection using Neural Activation Prior
Weilin Wan, Weizhong Zhang, Quan Zhou +2
Out-of-distribution detection (OOD) is a crucial technique for deploying machine learning models in the real world to handle the unseen scenarios. In this paper, we first propose a…