354 citations · 634 across the 70 of their papers we have counts for
38 papers · 1 filter
Merging Experts into One: Improving Computational Efficiency of Mixture of Experts
Shwai He, Run-Ze Fan, Liang Ding +3
Scaling the size of language models usually leads to remarkable advancements in NLP tasks. But it often comes with a price of growing computational cost. Although a sparse Mixture…
Efficient Federated Prompt Tuning for Black-box Large Pre-trained Models
Zihao Lin, Yan Sun, Yifan Shi +4
With the blowout development of pre-trained models (PTMs), the efficient tuning of these models for diverse downstream applications has emerged as a pivotal research concern. Altho…
Diversifying the Mixture-of-Experts Representation for Language Models with Orthogonal Optimizer
Boan Liu, Liang Ding, Li Shen +4
The Mixture of Experts (MoE) has emerged as a highly successful technique in deep learning, based on the principle of divide-and-conquer to maximize model capacity without signific…
Revisiting Plasticity in Visual Reinforcement Learning: Data, Modules and Training Stages
Guozheng Ma, Lu Li, Sen Zhang +6
Plasticity, the ability of a neural network to evolve with new data, is crucial for high-performance and sample-efficient visual reinforcement learning (VRL). Although methods like…
Towards Understanding Generalization and Stability Gaps between Centralized and Decentralized Federated Learning
Yan Sun, Li Shen, Dacheng Tao
As two mainstream frameworks in federated learning (FL), both centralized and decentralized approaches have shown great application value in practical scenarios. However, existing…
AdaMerging: Adaptive Model Merging for Multi-Task Learning
Enneng Yang, Zhenyi Wang, Li Shen +4
Multi-task learning (MTL) aims to empower a model to tackle multiple tasks simultaneously. A recent development known as task arithmetic has revealed that several models, each fine…
Unlikelihood Tuning on Negative Samples Amazingly Improves Zero-Shot Translation
Changtong Zan, Liang Ding, Li Shen +4
Zero-shot translation (ZST), which is generally based on a multilingual neural machine translation model, aims to translate between unseen language pairs in training data. The comm…
Are Large Language Models Really Robust to Word-Level Perturbations?
Haoyu Wang, Guozheng Ma, Cong Yu +10
The swift advancement in the scales and capabilities of Large Language Models (LLMs) positions them as promising tools for a variety of downstream tasks. In addition to the pursuit…
FedLALR: Client-Specific Adaptive Learning Rates Achieve Linear Speedup for Non-IID Data
Hao Sun, Li Shen, Shixiang Chen +4
Federated learning is an emerging distributed machine learning method, enables a large number of clients to train a model without exchanging their local data. The time cost of comm…
MerA: Merging Pretrained Adapters For Few-Shot Learning
Shwai He, Run-Ze Fan, Liang Ding +3
Adapter tuning, which updates only a few parameters, has become a mainstream method for fine-tuning pretrained language models to downstream tasks. However, it often yields subpar…
DFedADMM: Dual Constraints Controlled Model Inconsistency for Decentralized Federated Learning
Qinglun Li, Li Shen, Guanghao Li +2
To address the communication burden issues associated with federated learning (FL), decentralized federated learning (DFL) discards the central server and establishes a decentraliz…
Towards Understanding the Generalizability of Delayed Stochastic Gradient Descent
Xiaoge Deng, Li Shen, Shengwei Li +3
Stochastic gradient descent (SGD) performed in an asynchronous manner plays a crucial role in training large-scale machine learning models. However, the generalization performance…
Efficient Federated Learning via Local Adaptive Amended Optimizer with Linear Speedup
Yan Sun, Li Shen, Hao Sun +2
Adaptive optimization has achieved notable success for distributed learning while extending adaptive optimizer to federated Learning (FL) suffers from severe inefficiency, includin…
Systematic Investigation of Sparse Perturbed Sharpness-Aware Minimization Optimizer
Peng Mi, Li Shen, Tianhe Ren +6
Deep neural networks often suffer from poor generalization due to complex and non-convex loss landscapes. Sharpness-Aware Minimization (SAM) is a popular solution that smooths the…
Understanding How Consistency Works in Federated Learning via Stage-wise Relaxed Initialization
Yan Sun, Li Shen, Dacheng Tao
Federated learning (FL) is a distributed paradigm that coordinates massive local clients to collaboratively train a global model via stage-wise local training processes on the hete…
Instructed Diffuser with Temporal Condition Guidance for Offline Reinforcement Learning
Jifeng Hu, Yanchao Sun, Sili Huang +6
Recent works have shown the potential of diffusion models in computer vision and natural language processing. Apart from the classical supervised learning fields, diffusion models…
Towards More Suitable Personalization in Federated Learning via Decentralized Partial Model Training
Yifan Shi, Yingqi Liu, Yan Sun +4
Personalized federated learning (PFL) aims to produce the greatest personalized model for each client to face an insurmountable problem--data heterogeneity in real FL systems. Howe…
Prompt-Tuning Decision Transformer with Preference Ranking
Shengchao Hu, Li Shen, Ya Zhang +1
Prompt-tuning has emerged as a promising method for adapting pre-trained models to downstream tasks or aligning with human preferences. Prompt learning is widely used in NLP but ha…
Towards the Flatter Landscape and Better Generalization in Federated Learning under Client-level Differential Privacy
Yifan Shi, Kang Wei, Li Shen +4
To defend the inference attacks and mitigate the sensitive information leakages in Federated Learning (FL), client-level Differentially Private FL (DPFL) is the de-facto standard f…
Learning to Learn from APIs: Black-Box Data-Free Meta-Learning
Zixuan Hu, Li Shen, Zhenyi Wang +3
Data-free meta-learning (DFML) aims to enable efficient learning of new tasks by meta-learning from a collection of pre-trained models without access to the training data. Existing…
Learning Better with Less: Effective Augmentation for Sample-Efficient Visual Reinforcement Learning
Guozheng Ma, Linrui Zhang, Haoyu Wang +6
Data augmentation (DA) is a crucial technique for enhancing the sample efficiency of visual reinforcement learning (RL) algorithms. Notably, employing simple observation transforma…
Dynamic Regularized Sharpness Aware Minimization in Federated Learning: Approaching Global Consistency and Smooth Landscape
Yan Sun, Li Shen, Shixiang Chen +2
In federated learning (FL), a cluster of local clients are chaired under the coordination of the global server and cooperatively train one model with privacy protection. Due to the…
On Efficient Training of Large-Scale Deep Learning Models: A Literature Review
Li Shen, Yan Sun, Zhiyuan Yu +3
The field of deep learning has witnessed significant progress, particularly in computer vision (CV), natural language processing (NLP), and speech. The use of large-scale models tr…
Quantum Imitation Learning
Zhihao Cheng, Kaining Zhang, Li Shen +1
Despite remarkable successes in solving various complex decision-making tasks, training an imitation learning (IL) algorithm with deep neural networks (DNNs) suffers from the high…