TWEO: Transformers Without Extreme Outliers Enables FP8 Training And Quantization For Dummies

Guang Liang, Jie Shao, Ningyuan Tang +2

Native FP8 support in modern hardware is essential for training large Transformers, but is severely hindered by extreme activation outliers. Existing solutions either rely on compl…