BET: A Backtranslation Approach for Easy Data Augmentation in Transformer-based Paraphrase Identification Context

Abstract

Newly-introduced deep learning architectures, namely BERT, XLNet, RoBERTa and ALBERT, have been proved to be robust on several NLP tasks. However, the datasets trained on these architectures are fixed in terms of size and generalizability. To relieve this issue, we apply one of the most inexpensive solutions to update these datasets. We call this approach BET by which we analyze the backtranslation data augmentation on the transformer-based architectures. Using the Google Translate API with ten intermediary languages from ten different language families, we externally evaluate the results in the context of automatic paraphrase identification in a transformer-based framework. Our findings suggest that BET improves the paraphrase identification performance on the Microsoft Research Paraphrase Corpus (MRPC) to more than 3% on both accuracy and F1 score. We also analyze the augmentation in the low-data regime with downsampled versions of MRPC, Twitter Paraphrase Corpus (TPC) and Quora Question Pairs. In many low-data cases, we observe a switch from a failing model on the test set to reasonable performances. The results demonstrate that BET is a highly promising data augmentation technique: to push the current state-of-the-art of existing datasets and to bootstrap the utilization of deep learning architectures in the low-data regime of a hundred samples.

References in corpus (13)

• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
• Unsupervised Machine Translation Using Monolingual Corpora Only
• Data Augmentation using Pre-trained Transformer Models
• StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding
• ERNIE 2.0: A Continual Pre-training Framework for Language Understanding
• Learning Data Manipulation for Augmentation and Weighting
• XLDA: Cross-Lingual Data Augmentation for Natural Language Inference and Question Answering
• A Multi-cascaded Model with Data Augmentation for Enhanced Paraphrase Detection in Short Texts
• Text Data Augmentation Made Simple By Leveraging NLP Cloud APIs
• Not Enough Data? Deep Learning to the Rescue!
• E-Stitchup: Data Augmentation for Pre-Trained Embeddings
• Automatically Learning Data Augmentation Policies for Dialogue Tasks