Efficient Contrastive Learning via Novel Data Augmentation and Curriculum Learning

Abstract

We introduce EfficientCL, a memory-efficient continual pretraining method that applies contrastive learning with novel data augmentation and curriculum learning. For data augmentation, we stack two types of operation sequentially: cutoff and PCA jittering. While pretraining steps proceed, we apply curriculum learning by incrementing the augmentation degree for each difficulty step. After data augmentation is finished, contrastive learning is applied on projected embeddings of original and augmented examples. When finetuned on GLUE benchmark, our model outperforms baseline models, especially for sentence-level tasks. Additionally, this improvement is capable with only 70% of computational memory compared to the baseline model.

arXiv commentEMNLP 2021

References in corpus (7)

• A Simple Framework for Contrastive Learning of Visual Representations
• Big Self-Supervised Models are Strong Semi-Supervised Learners
• CLEAR: Contrastive Learning for Sentence Representation
• On The Power of Curriculum Learning in Training Deep Networks
• COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining
• A Simple but Tough-to-Beat Data Augmentation Approach for Natural Language Understanding and Generation
• CoDA: Contrast-enhanced and Diversity-promoting Data Augmentation for Natural Language Understanding