Pretraining Latent Information Feedback Transformers with Teacher Supervision

Dor Tirosh, Ido Amos, Mor Geva

Transformer language models (LMs) are feed-forward: deep-layer representations are never fed back to shallower layers, and the only pathway for information to flow downward across…