BERTs of a feather do not generalize together: Large variability in generalization across models with similar test set performance

R. Thomas McCoy, Junghyun Min, Tal Linzen

If the same neural network architecture is trained multiple times on the same dataset, will it make similar linguistic generalizations across runs? To study this question, we fine-…