SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimization

Abstract

Transfer learning has fundamentally changed the landscape of natural language processing (NLP) research. Many existing state-of-the-art models are first pre-trained on a large text corpus and then fine-tuned on downstream tasks. However, due to limited data resources from downstream tasks and the extremely large capacity of pre-trained models, aggressive fine-tuning often causes the adapted model to overfit the data of downstream tasks and forget the knowledge of the pre-trained model. To address the above issue in a more principled manner, we propose a new computational framework for robust and efficient fine-tuning for pre-trained language models. Specifically, our proposed framework contains two important ingredients: 1. Smoothness-inducing regularization, which effectively manages the capacity of the model; 2. Bregman proximal point optimization, which is a class of trust-region methods and can prevent knowledge forgetting. Our experiments demonstrate that our proposed method achieves the state-of-the-art performance on multiple NLP benchmarks.

arXiv commentThe 58th annual meeting of the Association for Computational Linguistics (ACL 2020)

Cited by in corpus (16)

• A Survey on Data Augmentation for Text Classification
• Is Neuro-Symbolic AI Meeting its Promise in Natural Language Processing? A Structured Review
• BERT-Deep CNN: State-of-the-Art for Sentiment Analysis of COVID-19 Tweets
• Low-rank Adaptation of Large Language Model Rescoring for Parameter-Efficient Speech Recognition
• Training-free Lexical Backdoor Attacks on Language Models
• Comparison between parameter-efficient techniques and full fine-tuning: A case study on multilingual news article classification
• FarsTail: A Persian Natural Language Inference Dataset
• Improving Pre-trained Language Model Fine-tuning with Noise Stability Regularization
• CSS-LM: A Contrastive Framework for Semi-supervised Fine-tuning of Pre-trained Language Models
• Infusing Finetuning with Semantic Dependencies
• Prompt Tuning with Soft Context Sharing for Vision-Language Models
• Leveraging Encoder-only Large Language Models for Mobile App Review Feature Extraction
• HateProof: Are Hateful Meme Detection Systems really Robust?
• Linear Transformations for Cross-lingual Sentiment Analysis
• Neural Architecture Search for Sentence Classification with BERT
• Unifying Economic and Language Models for Enhanced Sentiment Analysis of the Oil Market