2 papers
cs.LG2026
SwiftQK: Fast and Communication-Efficient Tensor Parallelism for Query-Key Normalization
Gyudong Kim, Wonjun Han, Young Geun Kim
Query-Key Normalization (QK-Norm) improves the training stability and quality of modern Large Language Models (LLMs). However, under Tensor Parallelism (TP), layerwise QK-Norm intr…
cs.LG2024
HeteroSwitch: Characterizing and Taming System-Induced Data Heterogeneity in Federated Learning
Gyudong Kim, Mehdi Ghasemi, Soroush Heidari +4
Federated Learning (FL) is a practical approach to train deep learning models collaboratively across user-end devices, protecting user privacy by retaining raw data on-device. In F…