5 papers
Weight Concentration Regularization for Improving Pruning Robustness Under High Sparsity
Vincent-Daniel Yun, Junhyuk Jo, Sunwoo Lee
Deep neural networks achieve outstanding performance across vision and language tasks, yet their large parameter counts limit deployment in resource-constrained settings. One-shot…
Biased Local SGD for Efficient Deep Learning on Heterogeneous Systems
Jihyun Lim, Junhyuk Jo, Chanhyeok Ko +3
Most parallel neural network training methods assume homogeneous computing resources. For example, synchronous data-parallel SGD suffers from significant synchronization overhead u…
Enabling Weak Client Participation via On-device Knowledge Distillation in Heterogeneous Federated Learning
Jihyun Lim, Junhyuk Jo, Tuo Zhang +1
Online Knowledge Distillation (KD) is recently highlighted to train large models in Federated Learning (FL) environments. Many existing studies adopt the logit ensemble method to p…
Layer-wise Adaptive Gradient Norm Penalizing Method for Efficient and Accurate Deep Learning
Sunwoo Lee
Sharpness-aware minimization (SAM) is known to improve the generalization performance of neural networks. However, it is not widely used in real-world applications yet due to its e…
Asynchronous Sharpness-Aware Minimization For Fast and Accurate Deep Learning
Junhyuk Jo, Jihyun Lim, Sunwoo Lee
Sharpness-Aware Minimization (SAM) is an optimization method that improves generalization performance of machine learning models. Despite its superior generalization, SAM has not b…