13 citations · 43 across the 14 of their papers we have counts for
14 papers
Output-Aware Rotation for INT2 KV-Cache Quantization
Vincent-Daniel Yun, Woosang Lim, Minsoo Cheong +4
The key-value (KV) cache has become a major memory and bandwidth bottleneck in long-context large language model inference, making ultra-low-bit quantization increasingly important…
Ghosted Layers: Unconstrained Activation Alignment for Recovering Layer-Pruned LLMs
Vincent-Daniel Yun, Junhyuk Jo, Sai Praneeth Karimireddy +1
Layer pruning removes entire Transformer decoder blocks from large language models, but introduces a mismatch between the hidden state received by the next surviving layer and the…
Rethinking Layer Redundancy: Calibration Matters More Than Search in LLM Depth Pruning
Minkyu Kim, Vincent-Daniel Yun, Youngrae Kim +3
Depth pruning improves the inference efficiency of large language models by removing Transformer blocks. Prior work typically treats layer redundancy as an inherent structural prop…
Locality-Aware Redundancy Pruning for LLM Depth Compression
Vincent-Daniel Yun, Youngrae Kim, Woosang Lim +3
Large language models are known to contain representational redundancy across network depth, making depth pruning an effective approach for improving inference efficiency. Existing…
Weight Concentration Regularization for Improving Pruning Robustness Under High Sparsity
Vincent-Daniel Yun, Junhyuk Jo, Sunwoo Lee
Deep neural networks achieve outstanding performance across vision and language tasks, yet their large parameter counts limit deployment in resource-constrained settings. One-shot…
Embracing Federated Learning: Enabling Weak Client Participation via Partial Model Training
Sunwoo Lee, Tuo Zhang, Saurav Prakash +2
In Federated Learning (FL), clients may have weak devices that cannot train the full model or even hold it in their memory space. To implement large-scale FL applications, thus, it…
mL-BFGS: A Momentum-based L-BFGS for Distributed Large-Scale Neural Network Optimization
Yue Niu, Zalan Fabian, Sunwoo Lee +2
Quasi-Newton methods still face significant challenges in training large-scale neural networks due to additional compute costs in the Hessian related computations and instability i…
TimelyFL: Heterogeneity-aware Asynchronous Federated Learning with Adaptive Partial Training
Tuo Zhang, Lei Gao, Sunwoo Lee +2
In cross-device Federated Learning (FL) environments, scaling synchronous FL methods is challenging as stragglers hinder the training process. Moreover, the availability of each cl…
FedML Parrot: A Scalable Federated Learning System via Heterogeneity-aware Scheduling on Sequential and Hierarchical Training
Zhenheng Tang, Xiaowen Chu, Ryan Yide Ran +7
Federated Learning (FL) enables collaborations among clients for train machine learning models while protecting their data privacy. Existing FL simulation platforms that are design…
FedAudio: A Federated Learning Benchmark for Audio Tasks
Tuo Zhang, Tiantian Feng, Samiul Alam +4
Federated learning (FL) has gained substantial attention in recent years due to the data privacy concerns related to the pervasiveness of consumer devices that continuously collect…
Federated Learning of Large Models at the Edge via Principal Sub-Model Training
Yue Niu, Saurav Prakash, Souvik Kundu +2
Federated Learning (FL) is emerging as a popular, promising decentralized learning framework that enables collaborative training among clients, with no need to share private data b…
Partial Model Averaging in Federated Learning: Performance Guarantees and Benefits
Sunwoo Lee, Anit Kumar Sahu, Chaoyang He +1
Local Stochastic Gradient Descent (SGD) with periodic model averaging (FedAvg) is a foundational algorithm in Federated Learning. The algorithm independently runs SGD on multiple w…
SSFL: Tackling Label Deficiency in Federated Learning via Personalized Self-Supervision
Chaoyang He, Zhengyu Yang, Erum Mushtaq +3
Federated Learning (FL) is transforming the ML training ecosystem from a centralized over-the-cloud setting to distributed training over edge devices in order to strengthen data pr…
Layer-wise Adaptive Model Aggregation for Scalable Federated Learning
Sunwoo Lee, Tuo Zhang, Chaoyang He +1
In Federated Learning, a common approach for aggregating local models across clients is periodic averaging of the full model parameters. It is, however, known that different layers…