Are Pre-trained Convolutions Better than Pre-trained Transformers?

Yi Tay, Mostafa Dehghani, Jai Gupta +4

In the era of pre-trained language models, Transformers are the de facto choice of model architectures. While recent research has shown promise in entirely convolutional, or CNN, a…