On the Runway Cascade of Transformers for Language Modeling

Hunjae Lee, Corey Clark

In decoder-only (causal) transformers, the computation graph created by causal masking routes information through both direct-path attention and indirect paths formed by intermedia…