Embracing the Dark Knowledge: Domain Generalization Using Regularized Knowledge Distillation

Abstract

Though convolutional neural networks are widely used in different tasks, lack of generalization capability in the absence of sufficient and representative data is one of the challenges that hinder their practical application. In this paper, we propose a simple, effective, and plug-and-play training strategy named Knowledge Distillation for Domain Generalization (KDDG) which is built upon a knowledge distillation framework with the gradient filter as a novel regularization term. We find that both the ``richer dark knowledge" from the teacher network, as well as the gradient filter we proposed, can reduce the difficulty of learning the mapping which further improves the generalization ability of the model. We also conduct experiments extensively to show that our framework can significantly improve the generalization capability of deep neural networks in different tasks including image classification, segmentation, reinforcement learning by comparing our method with existing state-of-the-art domain generalization techniques. Last but not the least, we propose to adopt two metrics to analyze our proposed method in order to better understand how our proposed method benefits the generalization capability of deep neural networks.

arXiv commentAccepted by ACM MM, 2021

References in corpus (21)

• Distilling the Knowledge in a Neural Network
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• On Calibration of Modern Neural Networks
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Understanding deep learning requires rethinking generalization
• Domain Generalization via Invariant Feature Representation
• Domain Generalization via Model-Agnostic Learning of Semantic Features
• Generalizing to Unseen Domains via Adversarial Data Augmentation
• Generalizing Across Domains via Cross-Gradient Training
• Invariant Risk Minimization
• Actor-Mimic: Deep Multitask and Transfer Reinforcement Learning
• Unifying distillation and privileged information
• Towards Understanding Knowledge Distillation
• Heterogeneous Domain Generalization via Domain Mixup
• Deep Model Compression: Distilling Knowledge from Noisy Teachers
• Domain Generalization for Medical Imaging Classification with Linear-Dependency Regularization
• Understanding and Improving Interpolation in Autoencoders via an Adversarial Regularizer
• Domain Generalization for Object Recognition with Multi-task Autoencoders
• Invariant Risk Minimization Games
• Explaining Knowledge Distillation by Quantifying the Knowledge
• Deep Domain-Adversarial Image Generation for Domain Generalisation