5 papers
Ghosted Layers: Unconstrained Activation Alignment for Recovering Layer-Pruned LLMs
Vincent-Daniel Yun, Junhyuk Jo, Sai Praneeth Karimireddy +1
Layer pruning removes entire Transformer decoder blocks from large language models, but introduces a mismatch between the hidden state received by the next surviving layer and the…
Weight Concentration Regularization for Improving Pruning Robustness Under High Sparsity
Vincent-Daniel Yun, Junhyuk Jo, Sunwoo Lee
Deep neural networks achieve outstanding performance across vision and language tasks, yet their large parameter counts limit deployment in resource-constrained settings. One-shot…
Biased Local SGD for Efficient Deep Learning on Heterogeneous Systems
Jihyun Lim, Junhyuk Jo, Chanhyeok Ko +3
Most parallel neural network training methods assume homogeneous computing resources. For example, synchronous data-parallel SGD suffers from significant synchronization overhead u…
Asynchronous Sharpness-Aware Minimization For Fast and Accurate Deep Learning
Junhyuk Jo, Jihyun Lim, Sunwoo Lee +1
Sharpness-Aware Minimization (SAM) is an optimization method that improves the generalization performance of machine learning models. Despite its effectiveness, it has not been wid…
Enabling Weak Client Participation via On-device Knowledge Distillation in Heterogeneous Federated Learning
Jihyun Lim, Junhyuk Jo, Tuo Zhang +1
Online Knowledge Distillation (KD) is recently highlighted to train large models in Federated Learning (FL) environments. Many existing studies adopt the logit ensemble method to p…