Simultaneous Learning of Trees and Representations for Extreme Classification and Density Estimation

Abstract

We consider multi-class classification where the predictor has a hierarchical structure that allows for a very large number of labels both at train and test time. The predictive power of such models can heavily depend on the structure of the tree, and although past work showed how to learn the tree structure, it expected that the feature vectors remained static. We provide a novel algorithm to simultaneously perform representation learning for the input data and learning of the hierarchi- cal predictor. Our approach optimizes an objec- tive function which favors balanced and easily- separable multi-way node partitions. We theoret- ically analyze this objective, showing that it gives rise to a boosting style property and a bound on classification error. We next show how to extend the algorithm to conditional density estimation. We empirically validate both variants of the al- gorithm on text classification and language mod- eling, respectively, and show that they compare favorably to common baselines in terms of accu- racy and running time.

References in corpus (10)

• YFCC100M: The New Data in Multimedia Research
• Ask Me Anything: Dynamic Memory Networks for Natural Language Processing
• Multi-Label Prediction via Compressed Sensing
• Efficient softmax approximation for GPUs
• Hierarchical Neural Language Models for Joint Representation of Streaming Documents and their Content
• An Exploration of Softmax Alternatives Belonging to the Spherical Loss Family
• Conditional Probability Tree Estimation Analysis and Algorithms
• Least Squares Revisited: Scalable Approaches for Multi-class Prediction
• Efficient Exact Gradient Update for training Deep Networks with Very Large Sparse Targets
• On the boosting ability of top-down decision tree learning algorithm for multiclass classification

Cited by in corpus (6)

• DeepXML: A Deep Extreme Multi-Label Learning Framework Applied to Short Text Documents
• ExKMC: Expanding Explainable $k$-Means Clustering
• Probabilistic Label Trees for Extreme Multi-label Classification
• sigmoidF1: A Smooth F1 Score Surrogate Loss for Multilabel Classification
• Unbiased scalable softmax optimization
• Top-K Pairwise Ranking: Bridging the Gap Among Ranking-Based Measures for Multi-Label Classification