Generalized Zero and Few-Shot Transfer for Facial Forgery Detection

Abstract

We propose Deep Distribution Transfer(DDT), a new transfer learning approach to address the problem of zero and few-shot transfer in the context of facial forgery detection. We examine how well a model (pre-)trained with one forgery creation method generalizes towards a previously unseen manipulation technique or different dataset. To facilitate this transfer, we introduce a new mixture model-based loss formulation that learns a multi-modal distribution, with modes corresponding to class categories of the underlying data of the source forgery method. Our core idea is to first pre-train an encoder neural network, which maps each mode of this distribution to the respective class labels, i.e., real or fake images in the source domain by minimizing wasserstein distance between them. In order to transfer this model to a new domain, we associate a few target samples with one of the previously trained modes. In addition, we propose a spatial mixup augmentation strategy that further helps generalization across domains. We find this learning strategy to be surprisingly effective at domain transfer compared to a traditional classification or even state-of-the-art domain adaptation/few-shot learning methods. For instance, compared to the best baseline, our method improves the classification accuracy by 4.88% for zero-shot and by 8.38% for the few-shot case transferred from the FaceForensics++ to Dessa dataset.

arXiv commentProject page: https://shivangi-aneja.github.io/ddt/

References in corpus (13)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Prototypical Networks for Few-shot Learning
• Deep Domain Confusion: Maximizing for Domain Invariance
• VisDA: The Visual Domain Adaptation Challenge
• Learning to Propagate Labels: Transductive Propagation Network for Few-shot Learning
• DeepFakes: a New Threat to Face Recognition? Assessment and Detection
• Few-Shot Learning with Graph Neural Networks
• ForensicTransfer: Weakly-supervised Domain Adaptation for Forgery Detection
• Automatic Face Reenactment
• The Deepfake Detection Challenge (DFDC) Preview Dataset
• DeeperForensics-1.0: A Large-Scale Dataset for Real-World Face Forgery Detection
• Self-Supervised Learning For Few-Shot Image Classification
• Reconstructing faces from voices

Cited by in corpus (3)

• MMSys'21 Grand Challenge on Detecting Cheapfakes
• COSMOS: Catching Out-of-Context Misinformation with Self-Supervised Learning
• Self-supervised GAN Detector