Taming Pretrained Transformers for Extreme Multi-label Text Classification

Abstract

We consider the extreme multi-label text classification (XMC) problem: given an input text, return the most relevant labels from a large label collection. For example, the input text could be a product description on Amazon.com and the labels could be product categories. XMC is an important yet challenging problem in the NLP community. Recently, deep pretrained transformer models have achieved state-of-the-art performance on many NLP tasks including sentence classification, albeit with small label sets. However, naively applying deep transformer models to the XMC problem leads to sub-optimal performance due to the large output space and the label sparsity issue. In this paper, we propose X-Transformer, the first scalable approach to fine-tuning deep transformer models for the XMC problem. The proposed method achieves new state-of-the-art results on four XMC benchmark datasets. In particular, on a Wiki dataset with around 0.5 million labels, the prec@1 of X-Transformer is 77.28%, a substantial improvement over state-of-the-art XMC approaches Parabel (linear) and AttentionXML (neural), which achieve 68.70% and 76.95% precision@1, respectively. We further apply X-Transformer to a product2query dataset from Amazon and gained 10.7% relative improvement on prec@1 over Parabel.

KDD 2020 Applied Data Track

References in corpus (14)

• Distributed Representations of Words and Phrases and their Compositionality
• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• Convolutional Sequence to Sequence Learning
• XLNet: Generalized Autoregressive Pretraining for Language Understanding
• Large-scale Multi-label Learning with Missing Labels
• Passage Re-ranking with BERT
• Multi-Label Prediction via Compressed Sensing
• DocBERT: BERT for Document Classification
• LSHTC: A Benchmark for Large-Scale Text Classification
• Pre-training Tasks for Embedding-based Large-scale Retrieval
• AttentionXML: Label Tree-based Attention-Aware Deep Model for High-Performance Extreme Multi-Label Text Classification
• A no-regret generalization of hierarchical softmax to extreme multi-label classification
• Stochastic Negative Mining for Learning with Large Output Spaces
• HAXMLNet: Hierarchical Attention Network for Extreme Multi-Label Text Classification

Cited by in corpus (9)

• Overview of BioASQ 2020: The eighth BioASQ challenge on Large-Scale Biomedical Semantic Indexing and Question Answering
• On Data Augmentation for Extreme Multi-label Classification
• Large Scale Legal Text Classification Using Transformer Models
• Data-Efficient Pretraining via Contrastive Self-Supervision
• GNN-XML: Graph Neural Networks for Extreme Multi-label Text Classification
• Extreme Multi-label Classification from Aggregated Labels
• Multilabel Classification by Hierarchical Partitioning and Data-dependent Grouping
• Improved Customer Transaction Classification using Semi-Supervised Knowledge Distillation
• Item Tagging for Information Retrieval: A Tripartite Graph Neural Network based Approach