3 papers
cs.LG2026
SwiftQK: Fast and Communication-Efficient Tensor Parallelism for Query-Key Normalization
Gyudong Kim, Wonjun Han, Young Geun Kim
Query-Key Normalization (QK-Norm) improves the training stability and quality of modern Large Language Models (LLMs). However, under Tensor Parallelism (TP), layerwise QK-Norm intr…
cs.LG2025
First Attentions Last: Better Exploiting First Attentions for Efficient Transformer Training
Gyudong Kim, Hyukju Na, Jin Hyeon Kim +6
As training billion-scale transformers becomes increasingly common, employing multiple distributed GPUs along with parallel training methods has become a standard practice. However…
cs.LG2024
HeteroSwitch: Characterizing and Taming System-Induced Data Heterogeneity in Federated Learning
Gyudong Kim, Mehdi Ghasemi, Soroush Heidari +4
Federated Learning (FL) is a practical approach to train deep learning models collaboratively across user-end devices, protecting user privacy by retaining raw data on-device. In F…