Does Your Optimizer Care How You Normalize? Normalization-Optimizer Coupling in LLM Training

Abdelrahman Abouzeid

In LLM training, normalization layers and optimizers are typically treated as independent design choices. In a 3x2 factorial at 1B parameters and 1000 training steps, we show this…