20 citations · 84 across the 25 of their papers we have counts for
25 papers
SimReg: Achieving Higher Performance in the Pretraining via Embedding Similarity Regularization
Yan Sun, Guoxia Wang, Jinle Zeng +6
Pretraining large language models (LLMs) with next-token prediction has led to remarkable advances, yet the context-dependent nature of token embeddings in such models results in h…
Rethinking the Personalized Relaxed Initialization in the Federated Learning: Consistency and Generalization
Li Shen, Yan Sun, Dacheng Tao
Federated learning (FL) is a distributed paradigm that coordinates massive local clients to collaboratively train a global model via stage-wise local training processes on the hete…
Multinoulli Extension: A Lossless Continuous Relaxation for Partition-Constrained Subset Selection
Qixin Zhang, Wei Huang, Yan Sun +3
Identifying the most representative subset for a close-to-submodular objective while satisfying the predefined partition constraint is a fundamental task with numerous applications…
Joint Selection for Large-Scale Pre-Training Data via Policy Gradient-based Mask Learning
Ziqing Fan, Yuqiao Xian, Yan Sun +1
A fine-grained data recipe is crucial for pre-training large language models, as it can significantly enhance training efficiency and model performance. One important ingredient in…
Effective Policy Learning for Multi-Agent Online Coordination Beyond Submodular Objectives
Qixin Zhang, Yan Sun, Can Jin +5
In this paper, we present two effective policy learning algorithms for multi-agent online coordination(MA-OC) problem. The first one, \texttt{MA-SPL}, not only can achieve the opti…
MaskPro: Linear-Space Probabilistic Learning for Strict (N:M)-Sparsity on LLMs
Yan Sun, Qixin Zhang, Zhiyuan Yu +3
The rapid scaling of large language models~(LLMs) has made inference efficiency a primary bottleneck in the practical deployment. To address this, semi-structured sparsity offers a…
A Survey on Mechanistic Interpretability for Multi-Modal Foundation Models
Zihao Lin, Samyadeep Basu, Mohammad Beigi +18
The rise of foundation models has transformed machine learning research, prompting efforts to uncover their inner workings and develop more efficient and reliable applications for…
TeZO: Empowering the Low-Rankness on the Temporal Dimension in the Zeroth-Order Optimization for Fine-tuning LLMs
Yan Sun, Tiansheng Huang, Liang Ding +2
Zeroth-order optimization (ZO) has demonstrated remarkable promise in efficient fine-tuning tasks for Large Language Models (LLMs). In particular, recent advances incorporate the l…
A Unified Analysis for Finite Weight Averaging
Peng Wang, Li Shen, Zerui Tao +3
Averaging iterations of Stochastic Gradient Descent (SGD) have achieved empirical success in training deep learning models, such as Stochastic Weight Averaging (SWA), Exponential M…
Stability and Generalization for Distributed SGDA
Miaoxi Zhu, Yan Sun, Li Shen +2
Minimax optimization is gaining increasing attention in modern machine learning applications. Driven by large-scale models and massive volumes of data collected from edge devices,…
A-FedPD: Aligning Dual-Drift is All Federated Primal-Dual Learning Needs
Yan Sun, Li Shen, Dacheng Tao
As a popular paradigm for juggling data privacy and collaborative training, federated learning (FL) is flourishing to distributively process the large scale of heterogeneous datase…
Convergent Differential Privacy Analysis for General Federated Learning
Yan Sun, Qixin Zhang, Li Shen +1
The powerful cooperation of federated learning (FL) and differential privacy~(DP) provides a promising paradigm for the large-scale private clients. However, existing analyses in F…
Efficient Federated Prompt Tuning for Black-box Large Pre-trained Models
Zihao Lin, Yan Sun, Yifan Shi +4
With the blowout development of pre-trained models (PTMs), the efficient tuning of these models for diverse downstream applications has emerged as a pivotal research concern. Altho…
Towards Understanding Generalization and Stability Gaps between Centralized and Decentralized Federated Learning
Yan Sun, Li Shen, Dacheng Tao
As two mainstream frameworks in federated learning (FL), both centralized and decentralized approaches have shown great application value in practical scenarios. However, existing…
Temporal Sentence Grounding in Streaming Videos
Tian Gan, Xiao Wang, Yan Sun +3
This paper aims to tackle a novel task - Temporal Sentence Grounding in Streaming Videos (TSGSV). The goal of TSGSV is to evaluate the relevance between a video stream and a given…
Efficient Federated Learning via Local Adaptive Amended Optimizer with Linear Speedup
Yan Sun, Li Shen, Hao Sun +2
Adaptive optimization has achieved notable success for distributed learning while extending adaptive optimizer to federated Learning (FL) suffers from severe inefficiency, includin…
Understanding How Consistency Works in Federated Learning via Stage-wise Relaxed Initialization
Yan Sun, Li Shen, Dacheng Tao
Federated learning (FL) is a distributed paradigm that coordinates massive local clients to collaboratively train a global model via stage-wise local training processes on the hete…
Towards More Suitable Personalization in Federated Learning via Decentralized Partial Model Training
Yifan Shi, Yingqi Liu, Yan Sun +4
Personalized federated learning (PFL) aims to produce the greatest personalized model for each client to face an insurmountable problem--data heterogeneity in real FL systems. Howe…
Dynamic Regularized Sharpness Aware Minimization in Federated Learning: Approaching Global Consistency and Smooth Landscape
Yan Sun, Li Shen, Shixiang Chen +2
In federated learning (FL), a cluster of local clients are chaired under the coordination of the global server and cooperatively train one model with privacy protection. Due to the…
On Efficient Training of Large-Scale Deep Learning Models: A Literature Review
Li Shen, Yan Sun, Zhiyuan Yu +3
The field of deep learning has witnessed significant progress, particularly in computer vision (CV), natural language processing (NLP), and speech. The use of large-scale models tr…
Visual Prompt Based Personalized Federated Learning
Guanghao Li, Wansen Wu, Yan Sun +3
As a popular paradigm of distributed learning, personalized federated learning (PFL) allows personalized models to improve generalization ability and robustness by utilizing knowle…
Subspace based Federated Unlearning
Guanghao Li, Li Shen, Yan Sun +3
Federated learning (FL) enables multiple clients to train a machine learning model collaboratively without exchanging their local data. Federated unlearning is an inverse FL proces…
Fusion of Global and Local Knowledge for Personalized Federated Learning
Tiansheng Huang, Li Shen, Yan Sun +2
Personalized federated learning, as a variant of federated learning, trains customized models for clients using their heterogeneously distributed data. However, it is still inconcl…
FedSpeed: Larger Local Interval, Less Communication Round, and Higher Generalization Accuracy
Yan Sun, Li Shen, Tiansheng Huang +2
Federated learning is an emerging distributed machine learning framework which jointly trains a global model via a large number of local devices with data privacy protections. Its…