68 citations · 135 across the 38 of their papers we have counts for
10 papers · 1 filter
Adapt-Pruner: Adaptive Structural Pruning for Efficient Small Language Model Training
Rui Pan, Shivanshu Shekhar, Boyao Wang +5
Small language models (SLMs) have attracted considerable attention from both academia and industry due to their broad range of applications in edge devices. To obtain SLMs with str…
ScaleBiO: Scalable Bilevel Optimization for LLM Data Reweighting
Rui Pan, Dylan Zhang, Hanning Zhang +6
Bilevel optimization has shown its utility across various machine learning settings, yet most algorithms in practice require second-order information, making it challenging to scal…
LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuning
Rui Pan, Xiang Liu, Shizhe Diao +4
The machine learning community has witnessed impressive advancements since large language models (LLMs) first appeared. Yet, their massive memory consumption has become a significa…
Mitigating the Alignment Tax of RLHF
Yong Lin, Hangyu Lin, Wei Xiong +14
LLMs acquire a wide range of abilities during pre-training, but aligning LLMs under Reinforcement Learning with Human Feedback (RLHF) can lead to forgetting pretrained abilities, w…
Effective Bilevel Optimization via Minimax Reformulation
Xiaoyu Wang, Rui Pan, Renjie Pi +1
Bilevel optimization has found successful applications in various machine learning problems, including hyper-parameter optimization, data cleaning, and meta-learning. However, its…
Probabilistic Bilevel Coreset Selection
Xiao Zhou, Renjie Pi, Weizhong Zhang +2
The goal of coreset selection in supervised learning is to produce a weighted subset of data, so that training only on the subset achieves similar performance as training on the en…
Model Agnostic Sample Reweighting for Out-of-Distribution Learning
Xiao Zhou, Yong Lin, Renjie Pi +4
Distributionally robust optimization (DRO) and invariant risk minimization (IRM) are two popular methods proposed to improve out-of-distribution (OOD) generalization performance of…
DYNAFED: Tackling Client Data Heterogeneity with Global Dynamics
Renjie Pi, Weizhong Zhang, Yueqi Xie +4
The Federated Learning (FL) paradigm is known to face challenges under heterogeneous client data. Local training on non-iid distributed data results in deflected local optimum, whi…
Robust Federated Learning against both Data Heterogeneity and Poisoning Attack via Aggregation Optimization
Yueqi Xie, Weizhong Zhang, Renjie Pi +4
Non-IID data distribution across clients and poisoning attacks are two main challenges in real-world federated learning (FL) systems. While both of them have attracted great resear…
Bridging the Gap between Sample-based and One-shot Neural Architecture Search with BONAS
Han Shi, Renjie Pi, Hang Xu +3
Neural Architecture Search (NAS) has shown great potentials in finding better neural network designs. Sample-based NAS is the most reliable approach which aims at exploring the sea…