How to Fine-Tune BERT for Text Classification?

Abstract

Language model pre-training has proven to be useful in learning universal language representations. As a state-of-the-art language model pre-training model, BERT (Bidirectional Encoder Representations from Transformers) has achieved amazing results in many language understanding tasks. In this paper, we conduct exhaustive experiments to investigate different fine-tuning methods of BERT on text classification task and provide a general solution for BERT fine-tuning. Finally, the proposed solution obtains new state-of-the-art results on eight widely-studied text classification datasets.

References in corpus (15)

• Distributed Representations of Words and Phrases and their Compositionality
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Distributed Representations of Sentences and Documents
• How transferable are features in deep neural networks?
• A Structured Self-attentive Sentence Embedding
• Recurrent Neural Network for Text Classification with Multi-Task Learning
• Skip-Thought Vectors
• Semi-supervised Sequence Learning
• GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks
• An efficient framework for learning sentence representations
• Multi-Task Deep Neural Networks for Natural Language Understanding
• Empower Sequence Labeling with Task-Aware Neural Language Model
• Generative and Discriminative Text Classification with Recurrent Neural Networks
• Neural Speed Reading via Skim-RNN
• Deconvolutional Latent-Variable Model for Text Sequence Matching

Cited by in corpus (29)

• A Benchmark Study of Machine Learning Models for Online Fake News Detection
• Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey
• FinBERT: Financial Sentiment Analysis with Pre-trained Language Models
• Advances of Machine Learning in Materials Science: Ideas and Techniques
• Well-calibrated Confidence Measures for Multi-label Text Classification with a Large Number of Labels
• A Survey of Active Learning for Text Classification using Deep Neural Networks
• Parsing with Multilingual BERT, a Small Corpus, and a Small Treebank
• BERT Goes to Law School: Quantifying the Competitive Advantage of Access to Large Legal Corpora in Contract Understanding
• Text Guide: Improving the quality of long text classification by a text selection method based on feature importance
• Multilingual Twitter Corpus and Baselines for Evaluating Demographic Bias in Hate Speech Recognition
• Language models and Automated Essay Scoring
• To Tune or Not To Tune? How About the Best of Both Worlds?
• The EarlyBIRD Catches the Bug: On Exploiting Early Layers of Encoder Models for More Efficient Code Classification
• MathBERT: A Pre-trained Language Model for General NLP Tasks in Mathematics Education
• Multi-Task Bidirectional Transformer Representations for Irony Detection
• COVID-19 Fake News Detection Using Bidirectional Encoder Representations from Transformers Based Models
• A Further Study of Unsupervised Pre-training for Transformer Based Speech Recognition
• Gradient-Based Adversarial Training on Transformer Networks for Detecting Check-Worthy Factual Claims
• Does Non-COVID19 Lung Lesion Help? Investigating Transferability in COVID-19 CT Image Segmentation
• Sentiment Analysis Using Averaged Weighted Word Vector Features
• Toward Tag-free Aspect Based Sentiment Analysis: A Multiple Attention Network Approach
• ZJUKLAB at SemEval-2021 Task 4: Negative Augmentation with Language Model for Reading Comprehension of Abstract Meaning
• Emergent Properties of Finetuned Language Representation Models
• NLPBK at VLSP-2020 shared task: Compose transformer pretrained models for Reliable Intelligence Identification on Social network
• General Purpose Text Embeddings from Pre-trained Language Models for Scalable Inference
• IIT_kgp at FinCausal 2020, Shared Task 1: Causality Detection using Sentence Embeddings in Financial Reports
• BERT based freedom to operate patent analysis
• PEL-BERT: A Joint Model for Protocol Entity Linking
• Graph Convolutional Network for Swahili News Classification