Central Moment Discrepancy (CMD) for Domain-Invariant Representation Learning

Abstract

The learning of domain-invariant representations in the context of domain adaptation with neural networks is considered. We propose a new regularization method that minimizes the discrepancy between domain-specific latent feature representations directly in the hidden activation space. Although some standard distribution matching approaches exist that can be interpreted as the matching of weighted sums of moments, e.g. Maximum Mean Discrepancy (MMD), an explicit order-wise matching of higher order moments has not been considered before. We propose to match the higher order central moments of probability distributions by means of order-wise moment differences. Our model does not require computationally expensive distance and kernel matrix computations. We utilize the equivalent representation of probability distributions by moment sequences to define a new distance function, called Central Moment Discrepancy (CMD). We prove that CMD is a metric on the set of probability distributions on a compact interval. We further prove that convergence of probability distributions on compact intervals w.r.t. the new metric implies convergence in distribution of the respective random variables. We test our approach on two different benchmark data sets for object recognition (Office) and sentiment analysis of product reviews (Amazon reviews). CMD achieves a new state-of-the-art performance on most domain adaptation tasks of Office and outperforms networks trained with MMD, Variational Fair Autoencoders and Domain Adversarial Neural Networks on Amazon reviews. In addition, a post-hoc parameter sensitivity analysis shows that the new approach is stable w.r.t. parameter changes in a certain interval. The source code of the experiments is publicly available.

arXiv commentExtended journal version published: https://doi.org/10.1016/j.ins.2019.01.025

References in corpus (10)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• ADADELTA: An Adaptive Learning Rate Method
• Learning Transferable Features with Deep Adaptation Networks
• Deep Domain Confusion: Maximizing for Domain Invariance
• Marginalized Denoising Autoencoders for Domain Adaptation
• Revisiting Batch Normalization For Practical Domain Adaptation
• Generative Moment Matching Networks
• Deep CORAL: Correlation Alignment for Deep Domain Adaptation
• Generative Models and Model Criticism via Optimized Maximum Mean Discrepancy
• FastMMD: Ensemble of Circular Discrepancy for Efficient Two-Sample Test

Cited by in corpus (69)

• Deep Visual Domain Adaptation: A Survey
• Deep Subdomain Adaptation Network for Image Classification
• MMD GAN: Towards Deeper Understanding of Moment Matching Network
• A Comprehensive Survey on Transfer Learning
• Unsupervised Domain Adaptation with Adversarial Residual Transform Networks
• Self-supervised Autoregressive Domain Adaptation for Time Series Data
• Generalized Domain Conditioned Adaptation Network
• FDA: Fourier Domain Adaptation for Semantic Segmentation
• Adversarial Domain Adaptation with Prototype-Based Normalized Output Conditioner
• Learning Target Domain Specific Classifier for Partial Domain Adaptation
• Domain-Symmetric Networks for Adversarial Domain Adaptation
• Unsupervised Domain Adaptation with Progressive Adaptation of Subspaces
• Multi-target Unsupervised Domain Adaptation without Exactly Shared Categories
• Feature Alignment and Restoration for Domain Generalization and Adaptation
• Transfer Learning with Dynamic Adversarial Adaptation Network
• Show, Adapt and Tell: Adversarial Training of Cross-domain Image Captioner
• Joint Domain Alignment and Discriminative Feature Learning for Unsupervised Deep Domain Adaptation
• Interpreting Deep Learning Models in Natural Language Processing: A Review
• Domain Adaptation without Source Data
• No More Discrimination: Cross City Adaptation of Road Scene Segmenters
• Shift-Robust GNNs: Overcoming the Limitations of Localized Graph Training Data
• Importance Weighted Adversarial Nets for Partial Domain Adaptation
• Refining Language Models with Compositional Explanations
• TWINs: Two Weighted Inconsistency-reduced Networks for Partial Domain Adaptation
• Cross-domain Contrastive Learning for Unsupervised Domain Adaptation
• Domain-Invariant Adversarial Learning for Unsupervised Domain Adaption
• AdaStereo: A Simple and Efficient Approach for Adaptive Stereo Matching
• ConDA: Continual Unsupervised Domain Adaptation
• Distributional Adversarial Networks
• Coupled Deep Learning for Heterogeneous Face Recognition
• Adversarial Discriminative Heterogeneous Face Recognition
• A Survey on Deep Domain Adaptation and Tiny Object Detection Challenges, Techniques and Datasets
• Dual Adversarial Domain Adaptation
• Inter-Region Affinity Distillation for Road Marking Segmentation
• Unsupervised Multi-Source Domain Adaptation for Person Re-Identification
• Style Normalization and Restitution for Domain Generalization and Adaptation
• Cross-Domain Gradient Discrepancy Minimization for Unsupervised Domain Adaptation
• HoMM: Higher-order Moment Matching for Unsupervised Domain Adaptation
• Learning Smooth Representation for Unsupervised Domain Adaptation
• Domain-Specific Suppression for Adaptive Object Detection
• Reducing the Covariate Shift by Mirror Samples in Cross Domain Alignment
• Selective Transfer with Reinforced Transfer Network for Partial Domain Adaptation
• BiPC: Bidirectional Probability Calibration for Unsupervised Domain Adaption
• Disentanglement for Discriminative Visual Recognition
• Towards Self-similarity Consistency and Feature Discrimination for Unsupervised Domain Adaptation
• Fine-grained Domain Adaptive Crowd Counting via Point-derived Segmentation
• Multiple Classifiers Based Maximum Classifier Discrepancy for Unsupervised Domain Adaptation
• Domain Adaptation for Sentiment Analysis Using Increased Intraclass Separation
• Learning Invariant Representation with Consistency and Diversity for Semi-supervised Source Hypothesis Transfer
• Multi-Anchor Active Domain Adaptation for Semantic Segmentation
• Improving Unsupervised Domain Adaptation by Reducing Bi-level Feature Redundancy
• Self-adaptive Re-weighted Adversarial Domain Adaptation
• Multi-Source Domain Adaptation for Text Classification via DistanceNet-Bandits
• Improved Hard Example Mining by Discovering Attribute-based Hard Person Identity
• Unsupervised Multi-Domain Multimodal Image-to-Image Translation with Explicit Domain-Constrained Disentanglement
• Compositional Models: Multi-Task Learning and Knowledge Transfer with Modular Networks
• Domain Impression: A Source Data Free Domain Adaptation Method
• Weighed Domain-Invariant Representation Learning for Cross-domain Sentiment Analysis
• Post-Hoc Domain Adaptation via Guided Data Homogenization
• Automatic Online Multi-Source Domain Adaptation
• ADeLA: Automatic Dense Labeling with Attention for Viewpoint Adaptation in Semantic Segmentation
• Leveraging Motion Priors in Videos for Improving Human Segmentation
• Teacher-Student Competition for Unsupervised Domain Adaptation
• DWMD: Dimensional Weighted Orderwise Moment Discrepancy for Domain-specific Hidden Representation Matching
• Ensemble Multi-Source Domain Adaptation with Pseudolabels
• Parameter Reference Loss for Unsupervised Domain Adaptation
• On-target Adaptation
• Neural Random Projection: From the Initial Task To the Input Similarity Problem
• Domain Adaptation by Maximizing Population Correlation with Neural Architecture Search