2 citations · 3 across the 15 of their papers we have counts for
10 papers · 1 filter
FedSLoP: Memory-Efficient Federated Learning with Low-Rank Gradient Projection
Yutong He, Zhengyang Huang, Jiahe Geng +1
Federated learning enables a population of clients to collaboratively train machine learning models without exchanging their raw data, but standard algorithms such as FedAvg suffer…
Mixture-of-Channels: Exploiting Sparse FFNs for Efficient LLMs Pre-Training and Inference
Tong Wu, Yutong He, Bin Wang +1
Large language models (LLMs) have demonstrated remarkable success across diverse artificial intelligence tasks, driven by scaling laws that correlate model size and training data w…
An All-Reduce Compatible Top-K Compressor for Communication-Efficient Distributed Learning
Chuyan Chen, Chenyang Ma, Zhangxin Li +3
Communication remains a central bottleneck in large-scale distributed machine learning, and gradient sparsification has emerged as a promising strategy to alleviate this challenge.…
MISA: Memory-Efficient LLMs Optimization with Module-wise Importance Sampling
Yuxi Liu, Renjia Deng, Yutong He +3
The substantial memory demands of pre-training and fine-tuning large language models (LLMs) require memory-efficient optimization algorithms. One promising approach is layer-wise o…
Greedy Low-Rank Gradient Compression for Distributed Learning with Convergence Guarantees
Chuyan Chen, Yutong He, Pengrui Li +2
Distributed optimization is pivotal for large-scale signal processing and machine learning, yet communication overhead remains a major bottleneck. Low-rank gradient compression, in…
TAH-QUANT: Effective Activation Quantization in Pipeline Parallelism over Slow Network
Guangxin He, Yuan Cao, Yutong He +4
Decentralized training of large language models offers the opportunity to pool computational resources across geographically distributed participants, but is often bottlenecked by…
CE-LoRA: Computation-Efficient LoRA Fine-Tuning for Language Models
Guanduo Chen, Yutong He, Yipeng Hu +2
Large Language Models (LLMs) demonstrate exceptional performance across various tasks but demand substantial computational resources even for fine-tuning computation. Although Low-…
Subspace Optimization for Large Language Models with Convergence Guarantees
Yutong He, Pengrui Li, Yipeng Hu +2
Subspace optimization algorithms, such as GaLore (Zhao et al., 2024), have gained attention for pre-training and fine-tuning large language models (LLMs) due to their memory effici…
Unbiased Compression Saves Communication in Distributed Optimization: When and How Much?
Yutong He, Xinmeng Huang, Kun Yuan
Communication compression is a common technique in distributed optimization that can alleviate communication overhead by transmitting compressed gradients and model parameters. How…
Lower Bounds and Accelerated Algorithms in Distributed Stochastic Optimization with Communication Compression
Yutong He, Xinmeng Huang, Yiming Chen +2
Communication compression is an essential strategy for alleviating communication overhead by reducing the volume of information exchanged between computing nodes in large-scale dis…