StagFormer: Time Staggering Transformer Decoding for RunningLayers In Parallel

Dylan Cutler, Arun Kandoor, Nishanth Dikkala +3

Decoding in a Transformer based language model is inherently sequential as a token's embedding needs to pass through all the layers in the network before the generation of the next…