FitNets: Hints for Thin Deep Nets

Abstract

While depth tends to improve network performances, it also makes gradient-based training more difficult since deeper networks tend to be more non-linear. The recently proposed knowledge distillation approach is aimed at obtaining small and fast-to-execute models, and it has shown that a student network could imitate the soft output of a larger teacher network or ensemble of networks. In this paper, we extend this idea to allow the training of a student that is deeper and thinner than the teacher, using not only the outputs but also the intermediate representations learned by the teacher as hints to improve the training process and final performance of the student. Because the student intermediate hidden layer will generally be smaller than the teacher's intermediate hidden layer, additional parameters are introduced to map the student hidden layer to the prediction of the teacher hidden layer. This allows one to train deeper students that can generalize better or run faster, a trade-off that is controlled by the chosen student capacity. For example, on CIFAR-10, a deep student network with almost 10.4 times less parameters outperforms a larger, state-of-the-art teacher network.

References in corpus (11)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Do Deep Nets Really Need to be Deep?
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• Going Deeper with Convolutions
• Theano: new features and speed improvements
• Deeply-Supervised Nets
• Compressing Deep Convolutional Networks using Vector Quantization
• Speeding up Convolutional Neural Networks with Low Rank Expansions
• Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
• Pylearn2: a machine learning research library
• Knowledge Matters: Importance of Prior Information for Optimization

Cited by in corpus (360)

• Knowledge Distillation: A Survey
• Grad-CAM++: Improved Visual Explanations for Deep Convolutional Networks
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• Densely Connected Convolutional Networks
• A Survey of Model Compression and Acceleration for Deep Neural Networks
• Object Detection in 20 Years: A Survey
• Data Augmentation using Random Image Cropping and Patching for Deep CNNs
• The History Began from AlexNet: A Comprehensive Survey on Deep Learning Approaches
• Born Again Neural Networks
• Well-Read Students Learn Better: On the Importance of Pre-training Compact Models
• Residual Networks of Residual Networks: Multilevel Residual Networks
• Like What You Like: Knowledge Distill via Neuron Selectivity Transfer
• Slimmable Neural Networks
• Adaptive Multi-Teacher Multi-level Knowledge Distillation
• Low-resolution Face Recognition in the Wild via Selective Knowledge Distillation
• Hardware and Software Optimizations for Accelerating Deep Neural Networks: Survey of Current Trends, Challenges, and the Road Ahead
• Data-Free Knowledge Distillation for Deep Neural Networks
• Improving Multi-hop Knowledge Base Question Answering by Learning Intermediate Supervision Signals
• Efficient Medical Image Segmentation Based on Knowledge Distillation
• Group Knowledge Transfer: Federated Learning of Large CNNs at the Edge
• Dynamic Convolutions: Exploiting Spatial Sparsity for Faster Inference
• Bringing AI To Edge: From Deep Learning's Perspective
• A Transformer-Based Model With Self-Distillation for Multimodal Emotion Recognition in Conversations
• XONN: XNOR-based Oblivious Deep Neural Network Inference
• Knowledge Guided Disambiguation for Large-Scale Scene Classification with Multi-Resolution CNNs
• Net2Net: Accelerating Learning via Knowledge Transfer
• Distilled Siamese Networks for Visual Tracking
• Label Refinery: Improving ImageNet Classification through Label Progression
• Ensemble Kalman Inversion: A Derivative-Free Technique For Machine Learning Tasks
• Enable Deep Learning on Mobile Devices: Methods, Systems, and Applications
• TinyBERT: Distilling BERT for Natural Language Understanding
• Edge Intelligence: Architectures, Challenges, and Applications
• N2N Learning: Network to Network Compression via Policy Gradient Reinforcement Learning
• Universal representations:The missing link between faces, text, planktons, and cat breeds
• The Importance of Skip Connections in Biomedical Image Segmentation
• Recurrent Neural Network Training with Dark Knowledge Transfer
• Dataset Condensation with Gradient Matching
• Data-Free Adversarial Distillation
• Deep Model Compression: Distilling Knowledge from Noisy Teachers
• Spot-adaptive Knowledge Distillation
• Focus: Querying Large Video Datasets with Low Latency and Low Cost
• CondenseNet: An Efficient DenseNet using Learned Group Convolutions
• DELTA: DEep Learning Transfer using Feature Map with Attention for Convolutional Networks
• MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices
• Iterative Machine Teaching
• Double Similarity Distillation for Semantic Image Segmentation
• Understanding and Improving Knowledge Distillation
• Dynamic Sampling Networks for Efficient Action Recognition in Videos
• Paraphrasing Complex Network: Network Compression via Factor Transfer
• Interleaved Group Convolutions for Deep Neural Networks
• Feature-map-level Online Adversarial Knowledge Distillation
• Applications and Techniques for Fast Machine Learning in Science
• SEED: Self-supervised Distillation For Visual Representation
• Comprehensive Attention Self-Distillation for Weakly-Supervised Object Detection
• DyNet: Dynamic Convolution for Accelerating Convolutional Neural Networks
• Knowledge Distillation in Generations: More Tolerant Teachers Educate Better Students
• Deep Learning with Low Precision by Half-wave Gaussian Quantization
• On Representation Knowledge Distillation for Graph Neural Networks
• Contrastive Representation Distillation
• Lifelong Teacher-Student Network Learning
• Training Skinny Deep Neural Networks with Iterative Hard Thresholding Methods
• MRI-based Alzheimer's disease prediction via distilling the knowledge in multi-modal data
• Compact recurrent neural networks for acoustic event detection on low-energy low-complexity platforms
• Improving Fast Segmentation With Teacher-student Learning
• FastBERT: a Self-distilling BERT with Adaptive Inference Time
• Lets keep it simple, Using simple architectures to outperform deeper and more complex architectures
• Efficient Convolutional Neural Networks for Depth-Based Multi-Person Pose Estimation
• Compacting Deep Neural Networks for Internet of Things: Methods and Applications
• Cross-dimensional transfer learning in medical image segmentation with deep learning
• Deep Predictive Coding Network for Object Recognition
• MEAL V2: Boosting Vanilla ResNet-50 to 80%+ Top-1 Accuracy on ImageNet without Tricks
• QKD: Quantization-aware Knowledge Distillation
• Stacked Generative Adversarial Networks
• A Survey on Symbolic Knowledge Distillation of Large Language Models
• Data Distillation: Towards Omni-Supervised Learning
• A Strategy for an Uncompromising Incremental Learner
• ExpandNets: Linear Over-parameterization to Train Compact Convolutional Networks
• ReSSL: Relational Self-Supervised Learning with Weak Augmentation
• Knowledge Adaptation: Teaching to Adapt
• BlockDrop: Dynamic Inference Paths in Residual Networks
• Orthogonal Gradient Descent for Continual Learning
• Graph-based Knowledge Distillation by Multi-head Attention Network
• Multi-Residual Networks: Improving the Speed and Accuracy of Residual Networks
• FaceNet2ExpNet: Regularizing a Deep Face Recognition Net for Expression Recognition
• Knowledge Distillation via Route Constrained Optimization
• Hydra: Preserving Ensemble Diversity for Model Distillation
• HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precision
• Variational Information Distillation for Knowledge Transfer
• Deep Convolutional Neural Networks with Merge-and-Run Mappings
• A Learning Framework for n-bit Quantized Neural Networks toward FPGAs
• Multi-Task Incremental Learning for Object Detection
• DiSMEC - Distributed Sparse Machines for Extreme Multi-label Classification
• Finding Competitive Network Architectures Within a Day Using UCT
• Multi-Task Zipping via Layer-wise Neuron Sharing
• SCSP: Spectral Clustering Filter Pruning with Soft Self-adaption Manners
• A Novel Weight-Shared Multi-Stage CNN for Scale Robustness
• Feature Fusion for Online Mutual Knowledge Distillation
• Rain O'er Me: Synthesizing real rain to derain with data distillation
• Avoiding Your Teacher's Mistakes: Training Neural Networks with Controlled Weak Supervision
• Transfer Learning for Speech and Language Processing
• Large-Scale Generative Data-Free Distillation
• A Systematic Survey of Deep Learning-based Single-Image Super-Resolution
• Deep Pyramidal Residual Networks
• Low-Resolution Object Recognition with Cross-Resolution Relational Contrastive Distillation
• Learning Privacy-Preserving Student Networks via Discriminative-Generative Distillation
• Improving Multi-Modal Learning with Uni-Modal Teachers
• Learning Lightweight Pedestrian Detector with Hierarchical Knowledge Distillation
• Cross Modal Distillation for Supervision Transfer
• Knowledge Distillation from Internal Representations
• FEED: Feature-level Ensemble for Knowledge Distillation
• Bi-Real Net: Enhancing the Performance of 1-bit CNNs With Improved Representational Capability and Advanced Training Algorithm
• Packing Sparse Convolutional Neural Networks for Efficient Systolic Array Implementations: Column Combining Under Joint Optimization
• Attention-Based Depth Distillation with 3D-Aware Positional Encoding for Monocular 3D Object Detection
• Conducting Credit Assignment by Aligning Local Representations
• Knowledge Concentration: Learning 100K Object Classifiers in a Single CNN
• Radio2Text: Streaming Speech Recognition Using mmWave Radio Signals
• Distilling Knowledge from Graph Convolutional Networks
• Self-Knowledge Distillation in Natural Language Processing
• Knowledge Adaptation for Efficient Semantic Segmentation
• Perceptron Theory Can Predict the Accuracy of Neural Networks
• ZeroQ: A Novel Zero Shot Quantization Framework
• Multi-Label Knowledge Distillation
• Ranking Distillation: Learning Compact Ranking Models With High Performance for Recommender System
• Tensorial Neural Networks: Generalization of Neural Networks and Application to Model Compression
• Optimizing speed/accuracy trade-off for person re-identification via knowledge distillation
• FrankenSplit: Efficient Neural Feature Compression with Shallow Variational Bottleneck Injection for Mobile Edge Computing
• Few Shot Network Compression via Cross Distillation
• Mixed Precision Low-bit Quantization of Neural Network Language Models for Speech Recognition
• MSD: Multi-Self-Distillation Learning via Multi-classifiers within Deep Neural Networks
• Knowing What, Where and When to Look: Efficient Video Action Modeling with Attention
• Explaining Knowledge Distillation by Quantifying the Knowledge
• Knowledge Projection for Deep Neural Networks
• Knowledge distillation via adaptive instance normalization
• Cross-Modality Attention with Semantic Graph Embedding for Multi-Label Classification
• I-BERT: Integer-only BERT Quantization
• Interpreting Deep Classifier by Visual Distillation of Dark Knowledge
• Intermediate Deep Feature Compression: the Next Battlefield of Intelligent Sensing
• Revisiting Knowledge Distillation via Label Smoothing Regularization
• Joint framework with deep feature distillation and adaptive focal loss for weakly supervised audio tagging and acoustic event detection
• Exploiting Errors for Efficiency: A Survey from Circuits to Algorithms
• KTAN: Knowledge Transfer Adversarial Network
• Towards Training Recurrent Neural Networks for Lifelong Learning
• Knowledge distillation using unlabeled mismatched images
• Distilling Word Embeddings: An Encoding Approach
• Amalgamating Knowledge towards Comprehensive Classification
• Comparing Kullback-Leibler Divergence and Mean Squared Error Loss in Knowledge Distillation
• Knowledge Transfer Pre-training
• AdderNet: Do We Really Need Multiplications in Deep Learning?
• Zero-Shot Knowledge Distillation from a Decision-Based Black-Box Model
• Learning Efficient Detector with Semi-supervised Adaptive Distillation
• Centripetal SGD for Pruning Very Deep Convolutional Networks with Complicated Structure
• KDLSQ-BERT: A Quantized Bert Combining Knowledge Distillation with Learned Step Size Quantization
• Feature Incay for Representation Regularization
• Maximizing Discrimination Capability of Knowledge Distillation with Energy Function
• Orchestrating the Development Lifecycle of Machine Learning-Based IoT Applications: A Taxonomy and Survey
• Age Group and Gender Estimation in the Wild with Deep RoR Architecture
• Knowledge Squeezed Adversarial Network Compression
• Learning to Steer by Mimicking Features from Heterogeneous Auxiliary Networks
• Modularized Morphing of Neural Networks
• Representation Stability as a Regularizer for Improved Text Analytics Transfer Learning
• More Grounded Image Captioning by Distilling Image-Text Matching Model
• MimicDet: Bridging the Gap Between One-Stage and Two-Stage Object Detection
• Deep cross-modality (MR-CT) educed distillation learning for cone beam CT lung tumor segmentation
• Multi-View Attention Transfer for Efficient Speech Enhancement
• Improving Multi-Person Pose Estimation using Label Correction
• PAMS: Quantized Super-Resolution via Parameterized Max Scale
• Improving the Interpretability of Deep Neural Networks with Knowledge Distillation
• Robustness-Reinforced Knowledge Distillation with Correlation Distance and Network Pruning
• Hint-Based Training for Non-Autoregressive Machine Translation
• PyNET-QxQ: An Efficient PyNET Variant for QxQ Bayer Pattern Demosaicing in CMOS Image Sensors
• Balanced Knowledge Distillation for Long-tailed Learning
• Knowledge Distillation via Weighted Ensemble of Teaching Assistants
• Fully-adaptive Feature Sharing in Multi-Task Networks with Applications in Person Attribute Classification
• TKD: Temporal Knowledge Distillation for Active Perception
• Share your Model instead of your Data: Privacy Preserving Mimic Learning for Ranking
• A Generalized Zero-Shot Quantization of Deep Convolutional Neural Networks via Learned Weights Statistics
• Cold Brew: Distilling Graph Node Representations with Incomplete or Missing Neighborhoods
• CUP: Cluster Pruning for Compressing Deep Neural Networks
• DistFL: Distribution-aware Federated Learning for Mobile Scenarios
• Dynamic Capacity Networks
• MetaDistiller: Network Self-Boosting via Meta-Learned Top-Down Distillation
• Smallify: Learning Network Size while Training
• Learning Versatile Convolution Filters for Efficient Visual Recognition
• Basic Level Categorization Facilitates Visual Object Recognition
• Generative Knowledge Transfer for Neural Language Models
• Leveraging Angular Distributions for Improved Knowledge Distillation
• Adaptive Instance Distillation for Object Detection in Autonomous Driving
• Transferring Knowledge Distillation for Multilingual Social Event Detection
• Fast End-to-End Speech Recognition via Non-Autoregressive Models and Cross-Modal Knowledge Transferring from BERT
• Improving Weakly Supervised Visual Grounding by Contrastive Knowledge Distillation
• Circumventing Outliers of AutoAugment with Knowledge Distillation
• PURSUhInT: In Search of Informative Hint Points Based on Layer Clustering for Knowledge Distillation
• Multiscale Hierarchical Convolutional Networks
• Full-Cycle Energy Consumption Benchmark for Low-Carbon Computer Vision
• The State of Knowledge Distillation for Classification
• Blending LSTMs into CNNs
• The Shallow End: Empowering Shallower Deep-Convolutional Networks through Auxiliary Outputs
• Mosaicking to Distill: Knowledge Distillation from Out-of-Domain Data
• Rethinking Implicit Neural Representations for Vision Learners
• CNN-based RGB-D Salient Object Detection: Learn, Select and Fuse
• On the Depth of Deep Neural Networks: A Theoretical View
• Spatial Likelihood Voting with Self-Knowledge Distillation for Weakly Supervised Object Detection
• Revisiting Knowledge Distillation for Object Detection
• A Light-weight Deep Human Activity Recognition Algorithm Using Multi-knowledge Distillation
• General Instance Distillation for Object Detection
• Cross-lingual Distillation for Text Classification
• ISP Distillation
• Efficient Contextualized Representation: Language Model Pruning for Sequence Labeling
• Computation-Efficient Knowledge Distillation via Uncertainty-Aware Mixup
• Subclass Knowledge Distillation with Known Subclass Labels
• On architectural choices in deep learning: From network structure to gradient convergence and parameter estimation
• 3D-to-2D Distillation for Indoor Scene Parsing
• ABC-KD: Attention-Based-Compression Knowledge Distillation for Deep Learning-Based Noise Suppression
• Streaming egocentric action anticipation: An evaluation scheme and approach
• KDPhys: An Attention Guided 3D to 2D Knowledge Distillation for Real-time Video-Based Physiological Measurement
• Ensemble-Compression: A New Method for Parallel Training of Deep Neural Networks
• MarginDistillation: distillation for margin-based softmax
• Revisiting Adversarial Robustness Distillation: Robust Soft Labels Make Student Better
• CL2R: Compatible Lifelong Learning Representations
• DARC: Differentiable ARchitecture Compression
• CrowdTransfer: Enabling Crowd Knowledge Transfer in AIoT Community
• Knowledge Distillation Under Ideal Joint Classifier Assumption
• Data-Free Knowledge Distillation with Soft Targeted Transfer Set Synthesis
• Knowledge distillation for optimization of quantized deep neural networks
• Multilinear Compressive Learning with Prior Knowledge
• Fully Learnable Group Convolution for Acceleration of Deep Neural Networks
• Simple Distillation Baselines for Improving Small Self-supervised Models
• Efficient Video Classification Using Fewer Frames
• Online Knowledge Distillation via Multi-branch Diversity Enhancement
• AlphaNet: Improved Training of Supernets with Alpha-Divergence
• A Survey of Mobile Computing for the Visually Impaired
• Spirit Distillation: A Model Compression Method with Multi-domain Knowledge Transfer
• Semantically-Conditioned Negative Samples for Efficient Contrastive Learning
• On Deep Representation Learning from Noisy Web Images
• Bidirectional Knowledge Reconfiguration for Lightweight Point Cloud Analysis
• One-bit Supervision for Image Classification
• Triplet Distillation for Deep Face Recognition
• Cross-View Policy Learning for Street Navigation
• Progressive Learning of Low-Precision Networks
• Correlation Congruence for Knowledge Distillation
• Real-time Policy Distillation in Deep Reinforcement Learning
• Data-Driven Compression of Convolutional Neural Networks
• Knowledge Distillation in Document Retrieval
• DiabDeep: Pervasive Diabetes Diagnosis based on Wearable Medical Sensors and Efficient Neural Networks
• Deep Semi-supervised Knowledge Distillation for Overlapping Cervical Cell Instance Segmentation
• Beyond Self-Supervision: A Simple Yet Effective Network Distillation Alternative to Improve Backbones
• ADD: Augmented Disentanglement Distillation Framework for Improving Stock Trend Forecasting
• Deep Neural Compression Via Concurrent Pruning and Self-Distillation
• Knowledge Distillation via Instance-level Sequence Learning
• Joint-DetNAS: Upgrade Your Detector with NAS, Pruning and Dynamic Distillation
• LNPT: Label-free Network Pruning and Training
• Reducing the gap between streaming and non-streaming Transducer-based ASR by adaptive two-stage knowledge distillation
• On Self-Distilling Graph Neural Network
• Active Subspace of Neural Networks: Structural Analysis and Universal Attacks
• Accelerating Large Scale Knowledge Distillation via Dynamic Importance Sampling
• Graph-Adaptive Pruning for Efficient Inference of Convolutional Neural Networks
• ShelfNet for Fast Semantic Segmentation
• SECS: Efficient Deep Stream Processing via Class Skew Dichotomy
• Recent Advances in Efficient Computation of Deep Convolutional Neural Networks
• Matching Guided Distillation
• One-bit Supervision for Image Classification: Problem, Solution, and Beyond
• Distilling Pixel-Wise Feature Similarities for Semantic Segmentation
• Multi-view Contrastive Learning for Online Knowledge Distillation
• Filter Pruning using Hierarchical Group Sparse Regularization for Deep Convolutional Neural Networks
• Automatic Inference of Cross-modal Connection Topologies for X-CNNs
• Universal Adder Neural Networks
• Training Deep Neural Networks for Wireless Sensor Networks Using Loosely and Weakly Labeled Images
• I Have Seen Enough: A Teacher Student Network for Video Classification Using Fewer Frames
• Adversarial Robustness for Unsupervised Domain Adaptation
• On the Orthogonality of Knowledge Distillation with Other Techniques: From an Ensemble Perspective
• Paying more attention to snapshots of Iterative Pruning: Improving Model Compression via Ensemble Distillation
• Efficient Object Embedding for Spliced Image Retrieval
• Efficient Crowd Counting via Structured Knowledge Transfer
• Voice2Mesh: Cross-Modal 3D Face Model Generation from Voices
• ThumbNet: One Thumbnail Image Contains All You Need for Recognition
• Oracle Teacher: Leveraging Target Information for Better Knowledge Distillation of CTC Models
• Improving Task-Agnostic BERT Distillation with Layer Mapping Search
• The Tensor Track VII: From Quantum Gravity to Artificial Intelligence
• PatchBatch: a Batch Augmented Loss for Optical Flow
• Exploring Effective Distillation of Self-Supervised Speech Models for Automatic Speech Recognition
• High Frequency Residual Learning for Multi-Scale Image Classification
• The Pupil Has Become the Master: Teacher-Student Model-Based Word Embedding Distillation with Ensemble Learning
• Multi-level Knowledge Distillation via Knowledge Alignment and Correlation
• MobileFAN: Transferring Deep Hidden Representation for Face Alignment
• Multimodal Knowledge Expansion
• Prune Your Model Before Distill It
• Embedding Principle in Depth for the Loss Landscape Analysis of Deep Neural Networks
• RAM-NAS: Resource-aware Multiobjective Neural Architecture Search Method for Robot Vision Tasks
• Knowledge Distillation For Recurrent Neural Network Language Modeling With Trust Regularization
• EEG aided boosting of single-lead ECG based sleep staging with Deep Knowledge Distillation
• RBUE: A ReLU-Based Uncertainty Estimation Method of Deep Neural Networks
• Pool of Experts: Realtime Querying Specialized Knowledge in Massive Neural Networks
• On The Distribution of Penultimate Activations of Classification Networks
• Robust Student Network Learning
• False Negative Distillation and Contrastive Learning for Personalized Outfit Recommendation
• EIGEN: Ecologically-Inspired GENetic Approach for Neural Network Structure Searching from Scratch
• Data-free Knowledge Distillation for Segmentation using Data-Enriching GAN
• Quantization Mimic: Towards Very Tiny CNN for Object Detection
• Digging Deeper into CRNN Model in Chinese Text Images Recognition
• Knowledge Transfer Based Fine-grained Visual Classification
• Label Augmentation via Time-based Knowledge Distillation for Financial Anomaly Detection
• Enhancing Data-Free Adversarial Distillation with Activation Regularization and Virtual Interpolation
• A New Training Framework for Deep Neural Network
• Complementary Relation Contrastive Distillation
• Differentiable Feature Aggregation Search for Knowledge Distillation
• Prime-Aware Adaptive Distillation
• Dynamic Knowledge Distillation for Black-box Hypothesis Transfer Learning
• LabelEnc: A New Intermediate Supervision Method for Object Detection
• SoFAr: Shortcut-based Fractal Architectures for Binary Convolutional Neural Networks
• Prune2Edge: A Multi-Phase Pruning Pipelines to Deep Ensemble Learning in IIoT
• Blind Adversarial Pruning: Balance Accuracy, Efficiency and Robustness
• CoCo DistillNet: a Cross-layer Correlation Distillation Network for Pathological Gastric Cancer Segmentation
• Hierarchical Auxiliary Learning
• Class-Discriminative CNN Compression
• Topology Distillation for Recommender System
• Hidden State Guidance: Improving Image Captioning using An Image Conditioned Autoencoder
• Channel Planting for Deep Neural Networks using Knowledge Distillation
• Reducing the Deployment-Time Inference Control Costs of Deep Reinforcement Learning Agents via an Asymmetric Architecture
• 4-Connected Shift Residual Networks
• Hands-on Guidance for Distilling Object Detectors
• EvDistill: Asynchronous Events to End-task Learning via Bidirectional Reconstruction-guided Cross-modal Knowledge Distillation
• Comb Convolution for Efficient Convolutional Architecture
• Towards Making Deep Transfer Learning Never Hurt
• Sequential Dynamic Decision Making with Deep Neural Nets on a Test-Time Budget
• Bridging the gap between Human Action Recognition and Online Action Detection
• Exploring Knowledge Distillation of a Deep Neural Network for Multi-Script identification
• Neighbourhood Distillation: On the benefits of non end-to-end distillation
• Student-Teacher Learning from Clean Inputs to Noisy Inputs
• Self-Supervised Adaptation for Video Super-Resolution
• DNN Quantization with Attention
• An Improving Framework of regularization for Network Compression
• BridgeNets: Student-Teacher Transfer Learning Based on Recursive Neural Networks and its Application to Distant Speech Recognition
• Long Short-Term Sample Distillation
• Knowledge Distillation By Sparse Representation Matching
• Dual Correction Strategy for Ranking Distillation in Top-N Recommender System
• Private Knowledge Transfer via Model Distillation with Generative Adversarial Networks
• Learning from Matured Dumb Teacher for Fine Generalization
• Surrogate-assisted Particle Swarm Optimisation for Evolving Variable-length Transferable Blocks for Image Classification
• Learning Fast Matching Models from Weak Annotations
• Is the Meta-Learning Idea Able to Improve the Generalization of Deep Neural Networks on the Standard Supervised Learning?
• Fixed-point Factorized Networks
• On educating machines
• Structure Learning of Deep Neural Networks with Q-Learning
• Sequenced-Replacement Sampling for Deep Learning
• AutoTinyBERT: Automatic Hyper-parameter Optimization for Efficient Pre-trained Language Models
• Learning Compatible Embeddings
• WeClick: Weakly-Supervised Video Semantic Segmentation with Click Annotations
• Reconciling Feature-Reuse and Overfitting in DenseNet with Specialized Dropout
• Sequentially Aggregated Convolutional Networks
• Improving Streaming Transformer Based ASR Under a Framework of Self-supervised Learning
• A Studious Approach to Semi-Supervised Learning
• Hardware-Software Codesign of Accurate, Multiplier-free Deep Neural Networks
• Partial to Whole Knowledge Distillation: Progressive Distilling Decomposed Knowledge Boosts Student Better
• Constrained-size Tensorflow Models for YouTube-8M Video Understanding Challenge
• SlimNets: An Exploration of Deep Model Compression and Acceleration
• Activation Map Adaptation for Effective Knowledge Distillation
• Self-supervised Knowledge Distillation Using Singular Value Decomposition
• Learning Realistic Patterns from Unrealistic Stimuli: Generalization and Data Anonymization
• Student Helping Teacher: Teacher Evolution via Self-Knowledge Distillation
• When in Doubt, Summon the Titans: Efficient Inference with Large Models