DocBERT: BERT for Document Classification

Abstract

We present, to our knowledge, the first application of BERT to document classification. A few characteristics of the task might lead one to think that BERT is not the most appropriate model: syntactic structures matter less for content categories, documents can often be longer than typical BERT input, and documents often have multiple labels. Nevertheless, we show that a straightforward classification model using BERT is able to achieve the state of the art across four popular datasets. To address the computational expense associated with BERT inference, we distill knowledge from BERT-large to small bidirectional LSTMs, reaching BERT-base parity on multiple datasets using 30x fewer parameters. The primary contribution of our paper is improved baselines that can provide the foundation for future work.

References in corpus (7)

• Scikit-learn: Machine Learning in Python
• Distilling the Knowledge in a Neural Network
• XLNet: Generalized Autoregressive Pretraining for Language Understanding
• Do Deep Nets Really Need to be Deep?
• Recent Trends in Deep Learning Based Natural Language Processing
• Distilling Task-Specific Knowledge from BERT into Simple Neural Networks
• Neural Network Models for Paraphrase Identification, Semantic Textual Similarity, Natural Language Inference, and Question Answering

Cited by in corpus (39)

• A Benchmark Study of Machine Learning Models for Online Fake News Detection
• DIET: Lightweight Language Understanding for Dialogue Systems
• Well-calibrated Confidence Measures for Multi-label Text Classification with a Large Number of Labels
• Med-BERT: pre-trained contextualized embeddings on large-scale structured electronic health records for disease prediction
• Personality Trait Detection Using Bagged SVM over BERT Word Embedding Ensembles
• Automated Quality Assessment of Cognitive Behavioral Therapy Sessions Through Highly Contextualized Language Representations
• Automated scholarly paper review: Concepts, technologies, and challenges
• Description Based Text Classification with Reinforcement Learning
• Phenotyping of Clinical Notes with Improved Document Classification Models Using Contextualized Neural Language Models
• Exploring Transformers in Natural Language Generation: GPT, BERT, and XLNet
• Long-length Legal Document Classification
• Enriching BERT with Knowledge Graph Embeddings for Document Classification
• Taming Pretrained Transformers for Extreme Multi-label Text Classification
• When Graph Convolution Meets Double Attention: Online Privacy Disclosure Detection with Multi-Label Text Classification
• COVID-19 Fake News Detection Using Bidirectional Encoder Representations from Transformers Based Models
• Protoformer: Embedding Prototypes for Transformers
• Lacking the embedding of a word? Look it up into a traditional dictionary
• Digital Twin Based Disaster Management System Proposal: DT-DMS
• Large-Scale News Classification using BERT Language Model: Spark NLP Approach
• Efficient Document Image Classification Using Region-Based Graph Neural Network
• Document Classification for COVID-19 Literature
• Sentiment analysis in tweets: an assessment study from classical to modern text representation models
• Robust Layout-aware IE for Visually Rich Documents with Pre-trained Language Models
• Enhancing Label Correlation Feedback in Multi-Label Text Classification via Multi-Task Learning
• Multilingual Pre-Trained Transformers and Convolutional NN Classification Models for Technical Domain Identification
• Classifying Long Clinical Documents with Pre-trained Transformers
• On-Device Document Classification using multimodal features
• Deep Anomaly Detection by Residual Adaptation
• Optimizing Memory Placement using Evolutionary Graph Reinforcement Learning
• Comparison of Czech Transformers on Text Classification Tasks
• Does a Hybrid Neural Network based Feature Selection Model Improve Text Classification?
• PyEuroVoc: A Tool for Multilingual Legal Document Classification with EuroVoc Descriptors
• Personalized Patent Claim Generation and Measurement
• PEL-BERT: A Joint Model for Protocol Entity Linking
• Event-Driven Learning of Systematic Behaviours in Stock Markets
• Corruption Is Not All Bad: Incorporating Discourse Structure into Pre-training via Corruption for Essay Scoring
• A System for Worldwide COVID-19 Information Aggregation
• Cascaded Semantic and Positional Self-Attention Network for Document Classification
• Gender prediction using limited Twitter Data