Structured Knowledge Distillation for Dense Prediction

Abstract

In this work, we consider transferring the structure information from large networks to compact ones for dense prediction tasks in computer vision. Previous knowledge distillation strategies used for dense prediction tasks often directly borrow the distillation scheme for image classification and perform knowledge distillation for each pixel separately, leading to sub-optimal performance. Here we propose to distill structured knowledge from large networks to compact networks, taking into account the fact that dense prediction is a structured prediction problem. Specifically, we study two structured distillation schemes: i) pair-wise distillation that distills the pair-wise similarities by building a static graph; and ii) holistic distillation that uses adversarial training to distill holistic knowledge. The effectiveness of our knowledge distillation approaches is demonstrated by experiments on three dense prediction tasks: semantic segmentation, depth estimation and object detection. Code is available at: https://git.io/StructKD

arXiv commentv1:10 pages cvpr2019 accepted; v2:15 pages for a journal version; Code is available at: https://github.com/irfanICMLL/structure_knowledge_distillation; fix typos

References in corpus (18)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Conditional Generative Adversarial Nets
• SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
• Generative Adversarial Text to Image Synthesis
• Depth Map Prediction from a Single Image using a Multi-Scale Deep Network
• DSSD : Deconvolutional Single Shot Detector
• Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer
• Do Deep Nets Really Need to be Deep?
• Going Deeper with Convolutions
• Learning Depth from Single Monocular Images Using Deep Convolutional Neural Fields
• ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentation
• Learning Deconvolution Network for Semantic Segmentation
• OCNet: Object Context Network for Scene Parsing
• Semantic Segmentation using Adversarial Networks
• Do Deep Convolutional Nets Really Need to be Deep and Convolutional?
• Improving Fast Segmentation With Teacher-student Learning

Cited by in corpus (7)

• Compacting Deep Neural Networks for Internet of Things: Methods and Applications
• Multiscale Deep Equilibrium Models
• Visual Tuning
• Improving Semantic Segmentation via Self-Training
• MobileFAN: Transferring Deep Hidden Representation for Face Alignment
• Self-Regulation for Semantic Segmentation
• Multi-Scale Aligned Distillation for Low-Resolution Detection