Segment-Based Attention Masking for GPTs

Shahar Katz, Liran Ringel, Yaniv Romano +1

Modern Language Models (LMs) owe much of their success to masked causal attention, the backbone of Generative Pre-Trained Transformer (GPT) models. Although GPTs can process the en…