Efficient pre-training objectives for Transformers

Luca Di Liello, Matteo Gabburo, Alessandro Moschitti

The Transformer architecture deeply changed the natural language processing, outperforming all previous state-of-the-art models. However, well-known Transformer models like BERT, R…