14 citations · 25 across the 8 of their papers we have counts for
6 papers · 1 filter
Structural Self-Supervised Objectives for Transformers
Luca Di Liello
This thesis focuses on improving the pre-training of natural language models using unsupervised raw data to make them more efficient and aligned with downstream applications. In th…
Context-Aware Transformer Pre-Training for Answer Sentence Selection
Luca Di Liello, Siddhant Garg, Alessandro Moschitti
Answer Sentence Selection (AS2) is a core component for building an accurate Question Answering pipeline. AS2 models rank a set of candidate sentences based on how likely they answ…
Effective Pre-Training Objectives for Transformer-based Autoencoders
Luca Di Liello, Matteo Gabburo, Alessandro Moschitti
In this paper, we study trade-offs between efficiency, cost and accuracy when pre-training Transformer encoders with different pre-training objectives. For this purpose, we analyze…
Pre-training Transformer Models with Sentence-Level Objectives for Answer Sentence Selection
Luca Di Liello, Siddhant Garg, Luca Soldaini +1
An important task for designing QA systems is answer sentence selection (AS2): selecting the sentence containing (or constituting) the answer to a question from a set of retrieved…
Paragraph-based Transformer Pre-training for Multi-Sentence Inference
Luca Di Liello, Siddhant Garg, Luca Soldaini +1
Inference tasks such as answer sentence selection (AS2) or fact verification are typically solved by fine-tuning transformer-based models as individual sentence-pair classifiers. R…
Efficient pre-training objectives for Transformers
Luca Di Liello, Matteo Gabburo, Alessandro Moschitti
The Transformer architecture deeply changed the natural language processing, outperforming all previous state-of-the-art models. However, well-known Transformer models like BERT, R…