Data-Efficient Pretraining via Contrastive Self-Supervision

Nils Rethmeier, Isabelle Augenstein

For natural language processing `text-to-text' tasks, the prevailing approaches heavily rely on pretraining large self-supervised models on increasingly larger `task-external' data…