Mask-Based Priors Are More Persistent than Query-Key Initializations

Mingze Ma, Hemanth Saratchandran, Cameron Gordon +1

Transformers do not merely lack data on some Boolean extrapolation tasks; they generalize in a systematically wrong way. Recent work on generalization on the unseen has shown that,…