Strengthening Structural Inductive Biases by Pre-training to Perform Syntactic Transformations

Matthias Lindemann, Alexander Koller, Ivan Titov

Models need appropriate inductive biases to effectively learn from small amounts of data and generalize systematically outside of the training distribution. While Transformers are…