Improving Pretrained Models for Zero-shot Multi-label Text Classification through Reinforced Label Hierarchy Reasoning
arXiv:2104.01666
Abstract
Exploiting label hierarchies has become a promising approach to tackling the zero-shot multi-label text classification (ZS-MTC) problem. Conventional methods aim to learn a matching model between text and labels, using a graph encoder to incorporate label hierarchies to obtain effective label representations \cite{rios2018few}. More recently, pretrained models like BERT \cite{devlin2018bert} have been used to convert classification tasks into a textual entailment task \cite{yin-etal-2019-benchmarking}. This approach is naturally suitable for the ZS-MTC task. However, pretrained models are underexplored in the existing work because they do not generate individual vector representations for text or labels, making it unintuitive to combine them with conventional graph encoding methods. In this paper, we explore to improve pretrained models with label hierarchies on the ZS-MTC task. We propose a Reinforced Label Hierarchy Reasoning (RLHR) approach to encourage interdependence among labels in the hierarchies during training. Meanwhile, to overcome the weakness of flat predictions, we design a rollback algorithm that can remove logical errors from predictions during inference. Experimental results on three real-life datasets show that our approach achieves better performance and outperforms previous non-pretrained methods on the ZS-MTC task.
arXiv commentAccepted to Main Conference of NAACL 2021 as Long Paper
References in corpus (8)
- Adam: A Method for Stochastic Optimization
- HDLTex: Hierarchical Deep Learning for Text Classification
- Multi-Stage Document Ranking with BERT
- Understanding the Behaviors of BERT in Ranking
- ML-Net: multi-label classification of biomedical texts with deep neural networks
- LSHTC: A Benchmark for Large-Scale Text Classification
- Hierarchical Text Classification with Reinforced Label Assignment
- Universal Text Representation from BERT: An Empirical Study