Inducing Systematicity in Transformers by Attending to Structurally Quantized Embeddings

Yichen Jiang, Xiang Zhou, Mohit Bansal

Transformers generalize to novel compositions of structures and entities after being trained on a complex dataset, but easily overfit on datasets of insufficient complexity. We obs…