Fast Multi-Resolution Transformer Fine-tuning for Extreme Multi-label Text Classification
arXiv:2110.00685
Abstract
Extreme multi-label text classification (XMC) seeks to find relevant labels from an extreme large label collection for a given text input. Many real-world applications can be formulated as XMC problems, such as recommendation systems, document tagging and semantic search. Recently, transformer based XMC methods, such as X-Transformer and LightXML, have shown significant improvement over other XMC methods. Despite leveraging pre-trained transformer models for text representation, the fine-tuning procedure of transformer models on large label space still has lengthy computational time even with powerful GPUs. In this paper, we propose a novel recursive approach, XR-Transformer to accelerate the procedure through recursively fine-tuning transformer models on a series of multi-resolution objectives related to the original XMC objective function. Empirical results show that XR-Transformer takes significantly less training time compared to other transformer-based XMC models while yielding better state-of-the-art results. In particular, on the public Amazon-3M dataset with 3 million labels, XR-Transformer is not only 20x faster than X-Transformer but also improves the Precision@1 from 51% to 54%.
References in corpus (11)
- XLNet: Generalized Autoregressive Pretraining for Language Understanding
- Bonsai -- Diverse and Shallow Trees for Extreme Multi-label Classification
- Pre-training Tasks for Embedding-based Large-scale Retrieval
- AttentionXML: Label Tree-based Attention-Aware Deep Model for High-Performance Extreme Multi-Label Text Classification
- DeepXML: A Deep Extreme Multi-Label Learning Framework Applied to Short Text Documents
- Logarithmic Time Online Multiclass prediction
- DECAF: Deep Extreme Classification with Label Features
- PECOS: Prediction for Enormous and Correlated Output Spaces
- Logarithmic Time One-Against-Some
- LightXML: Transformer with Dynamic Negative Sampling for High-Performance Extreme Multi-label Text Classification
- Probabilistic Label Trees for Extreme Multi-label Classification