Transformers with Selective Access to Early Representations

Skye Gunasekaran, Téa Wright, Rui-Jie Zhu +1

Several recent Transformer architectures expose later layers to representations computed in the earliest layers, motivated by the observation that low-level features can become har…