SwiftQK: Fast and Communication-Efficient Tensor Parallelism for Query-Key Normalization

Gyudong Kim, Wonjun Han, Young Geun Kim

Query-Key Normalization (QK-Norm) improves the training stability and quality of modern Large Language Models (LLMs). However, under Tensor Parallelism (TP), layerwise QK-Norm intr…