68 citations · 501 across the 111 of their papers we have counts for
47 papers · 1 filter
Efficient Federated Prompt Tuning for Black-box Large Pre-trained Models
Zihao Lin, Yan Sun, Yifan Shi +4
With the blowout development of pre-trained models (PTMs), the efficient tuning of these models for diverse downstream applications has emerged as a pivotal research concern. Altho…
Learning from models beyond fine-tuning
Hongling Zheng, Li Shen, Anke Tang +5
Foundation models (FM) have demonstrated remarkable performance across a wide range of tasks (especially in the fields of natural language processing and computer vision), primaril…
Revisiting Plasticity in Visual Reinforcement Learning: Data, Modules and Training Stages
Guozheng Ma, Lu Li, Sen Zhang +6
Plasticity, the ability of a neural network to evolve with new data, is crucial for high-performance and sample-efficient visual reinforcement learning (VRL). Although methods like…
Parameter Efficient Multi-task Model Fusion with Partial Linearization
Anke Tang, Li Shen, Yong Luo +5
Large pre-trained models have enabled significant advances in machine learning and served as foundation components. Model fusion methods, such as task arithmetic, have been proven…
Towards Understanding Generalization and Stability Gaps between Centralized and Decentralized Federated Learning
Yan Sun, Li Shen, Dacheng Tao
As two mainstream frameworks in federated learning (FL), both centralized and decentralized approaches have shown great application value in practical scenarios. However, existing…
AdaMerging: Adaptive Model Merging for Multi-Task Learning
Enneng Yang, Zhenyi Wang, Li Shen +4
Multi-task learning (MTL) aims to empower a model to tackle multiple tasks simultaneously. A recent development known as task arithmetic has revealed that several models, each fine…
Are Large Language Models Really Robust to Word-Level Perturbations?
Haoyu Wang, Guozheng Ma, Cong Yu +10
The swift advancement in the scales and capabilities of Large Language Models (LLMs) positions them as promising tools for a variety of downstream tasks. In addition to the pursuit…
FedLALR: Client-Specific Adaptive Learning Rates Achieve Linear Speedup for Non-IID Data
Hao Sun, Li Shen, Shixiang Chen +4
Federated learning is an emerging distributed machine learning method, enables a large number of clients to train a model without exchanging their local data. The time cost of comm…
Continual Learning From a Stream of APIs
Enneng Yang, Zhenyi Wang, Li Shen +5
Continual learning (CL) aims to learn new tasks without forgetting previous tasks. However, existing CL methods require a large amount of raw data, which is often unavailable due t…
DFedADMM: Dual Constraints Controlled Model Inconsistency for Decentralized Federated Learning
Qinglun Li, Li Shen, Guanghao Li +2
To address the communication burden issues associated with federated learning (FL), decentralized federated learning (DFL) discards the central server and establishes a decentraliz…
Towards Understanding the Generalizability of Delayed Stochastic Gradient Descent
Xiaoge Deng, Li Shen, Shengwei Li +3
Stochastic gradient descent (SGD) performed in an asynchronous manner plays a crucial role in training large-scale machine learning models. However, the generalization performance…
Efficient Federated Learning via Local Adaptive Amended Optimizer with Linear Speedup
Yan Sun, Li Shen, Hao Sun +2
Adaptive optimization has achieved notable success for distributed learning while extending adaptive optimizer to federated Learning (FL) suffers from severe inefficiency, includin…
Heterogeneous Federated Learning: State-of-the-art and Research Challenges
Mang Ye, Xiuwen Fang, Bo Du +2
Federated learning (FL) has drawn increasing attention owing to its potential use in large-scale industrial applications. Existing federated learning works mainly focus on model ho…
Systematic Investigation of Sparse Perturbed Sharpness-Aware Minimization Optimizer
Peng Mi, Li Shen, Tianhe Ren +6
Deep neural networks often suffer from poor generalization due to complex and non-convex loss landscapes. Sharpness-Aware Minimization (SAM) is a popular solution that smooths the…
Structured Cooperative Learning with Graphical Model Priors
Shuangtong Li, Tianyi Zhou, Xinmei Tian +1
We study how to train personalized models for different tasks on decentralized devices with limited local data. We propose "Structured Cooperative Learning (SCooL)", in which a coo…
Understanding How Consistency Works in Federated Learning via Stage-wise Relaxed Initialization
Yan Sun, Li Shen, Dacheng Tao
Federated learning (FL) is a distributed paradigm that coordinates massive local clients to collaboratively train a global model via stage-wise local training processes on the hete…
Instructed Diffuser with Temporal Condition Guidance for Offline Reinforcement Learning
Jifeng Hu, Yanchao Sun, Sili Huang +6
Recent works have shown the potential of diffusion models in computer vision and natural language processing. Apart from the classical supervised learning fields, diffusion models…
Decentralized SGD and Average-direction SAM are Asymptotically Equivalent
Tongtian Zhu, Fengxiang He, Kaixuan Chen +2
Decentralized stochastic gradient descent (D-SGD) allows collaborative learning on massive devices simultaneously without the control of a central server. However, existing theorie…
Towards More Suitable Personalization in Federated Learning via Decentralized Partial Model Training
Yifan Shi, Yingqi Liu, Yan Sun +4
Personalized federated learning (PFL) aims to produce the greatest personalized model for each client to face an insurmountable problem--data heterogeneity in real FL systems. Howe…
Improving Heterogeneous Model Reuse by Density Estimation
Anke Tang, Yong Luo, Han Hu +5
This paper studies multiparty learning, aiming to learn a model using the private data of different participants. Model reuse is a promising solution for multiparty learning, assum…
Prompt-Tuning Decision Transformer with Preference Ranking
Shengchao Hu, Li Shen, Ya Zhang +1
Prompt-tuning has emerged as a promising method for adapting pre-trained models to downstream tasks or aligning with human preferences. Prompt learning is widely used in NLP but ha…
Towards the Flatter Landscape and Better Generalization in Federated Learning under Client-level Differential Privacy
Yifan Shi, Kang Wei, Li Shen +4
To defend the inference attacks and mitigate the sensitive information leakages in Federated Learning (FL), client-level Differentially Private FL (DPFL) is the de-facto standard f…
Learning to Learn from APIs: Black-Box Data-Free Meta-Learning
Zixuan Hu, Li Shen, Zhenyi Wang +3
Data-free meta-learning (DFML) aims to enable efficient learning of new tasks by meta-learning from a collection of pre-trained models without access to the training data. Existing…
Learning Better with Less: Effective Augmentation for Sample-Efficient Visual Reinforcement Learning
Guozheng Ma, Linrui Zhang, Haoyu Wang +6
Data augmentation (DA) is a crucial technique for enhancing the sample efficiency of visual reinforcement learning (RL) algorithms. Notably, employing simple observation transforma…