First Attentions Last: Better Exploiting First Attentions for Efficient Transformer Training

Gyudong Kim, Hyukju Na, Jin Hyeon Kim +6

As training billion-scale transformers becomes increasingly common, employing multiple distributed GPUs along with parallel training methods has become a standard practice. However…