Learning Deep Representations with Probabilistic Knowledge Transfer
arXiv:1803.10837
Abstract
Knowledge Transfer (KT) techniques tackle the problem of transferring the knowledge from a large and complex neural network into a smaller and faster one. However, existing KT methods are tailored towards classification tasks and they cannot be used efficiently for other representation learning tasks. In this paper a novel knowledge transfer technique, that is capable of training a student model that maintains the same amount of mutual information between the learned representation and a set of (possible unknown) labels as the teacher model, is proposed. Apart from outperforming existing KT techniques, the proposed method allows for overcoming several limitations of existing methods providing new insight into KT as well as novel KT applications, ranging from knowledge transfer from handcrafted feature extractors to {cross-modal} KT from the textual modality into the representation extracted from the visual modality of the data.
arXiv commentAccepted at ECCV2018
References in corpus (13)
- Adam: A Method for Stochastic Optimization
- Distilling the Knowledge in a Neural Network
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
- Binarized Neural Networks
- YOLO9000: Better, Faster, Stronger
- Bayesian Dark Knowledge
- MS-Celeb-1M: A Dataset and Benchmark for Large-Scale Face Recognition
- Net2Net: Accelerating Learning via Knowledge Transfer
- Recurrent Neural Network Training with Dark Knowledge Transfer
- DSOD: Learning Deeply Supervised Object Detectors from Scratch
- Transferring Knowledge from a RNN to a DNN
- Knowledge Transfer Pre-training