Beyond One-hot Encoding: lower dimensional target embedding

Abstract

Target encoding plays a central role when learning Convolutional Neural Networks. In this realm, One-hot encoding is the most prevalent strategy due to its simplicity. However, this so widespread encoding schema assumes a flat label space, thus ignoring rich relationships existing among labels that can be exploited during training. In large-scale datasets, data does not span the full label space, but instead lies in a low-dimensional output manifold. Following this observation, we embed the targets into a low-dimensional space, drastically improving convergence speed while preserving accuracy. Our contribution is two fold: (i) We show that random projections of the label space are a valid tool to find such lower dimensional embeddings, boosting dramatically convergence rates at zero computational cost; and (ii) we propose a normalized eigenrepresentation of the class manifold that encodes the targets with minimal information loss, improving the accuracy of random projections encoding while enjoying the same convergence rates. Experiments on CIFAR-100, CUB200-2011, Imagenet, and MIT Places demonstrate that the proposed approach drastically improves convergence speed while reaching very competitive accuracy rates.

Published at Image and Vision Computing

References in corpus (6)

• Efficient Estimation of Word Representations in Vector Space
• Caffe: Convolutional Architecture for Fast Feature Embedding
• Label-Embedding for Image Classification
• Multi-Label Prediction via Compressed Sensing
• HD-CNN: Hierarchical Deep Convolutional Neural Network for Large Scale Visual Recognition
• Deep Networks With Large Output Spaces

Cited by in corpus (14)

• Regularized target encoding outperforms traditional methods in supervised machine learning with high cardinality features
• URANUS: Radio Frequency Tracking, Classification and Identification of Unmanned Aircraft Vehicles
• Comprehensive Mapping of Continuous/Switching Circuits in CCM and DCM to Machine Learning Domain using Homogeneous Graph Neural Networks
• Hybrid Machine Learning techniques in the management of harmful algal blooms impact
• Fairness Implications of Encoding Protected Categorical Attributes
• Improving Perceptual Quality by Phone-Fortified Perceptual Loss using Wasserstein Distance for Speech Enhancement
• Predicting the Travel Distance of Patients to Access Healthcare using Deep Neural Networks
• Sequential Minimal Optimization Algorithm for One-Class Support Vector Machines With Privileged Information
• PairConnect: A Compute-Efficient MLP Alternative to Attention
• Multi-way Encoding for Robustness
• WisdomNet: Prognosis of COVID-19 with Slender Prospect of False Negative Cases and Vaticinating the Probability of Maturation to ARDS using Posteroanterior Chest X-Rays
• Beyond one-hot encoding? Journey into compact encoding for large multi-class segmentation
• Exploration of stable atomic configurations in graphene-like BCN systems by Bayesian optimization
• MUTE: Data-Similarity Driven Multi-hot Target Encoding for Neural Network Design