FakeCatcher: Detection of Synthetic Portrait Videos using Biological Signals

Abstract

The recent proliferation of fake portrait videos poses direct threats on society, law, and privacy. Believing the fake video of a politician, distributing fake pornographic content of celebrities, fabricating impersonated fake videos as evidence in courts are just a few real world consequences of deep fakes. We present a novel approach to detect synthetic content in portrait videos, as a preventive solution for the emerging threat of deep fakes. In other words, we introduce a deep fake detector. We observe that detectors blindly utilizing deep learning are not effective in catching fake content, as generative models produce formidably realistic results. Our key assertion follows that biological signals hidden in portrait videos can be used as an implicit descriptor of authenticity, because they are neither spatially nor temporally preserved in fake content. To prove and exploit this assertion, we first engage several signal transformations for the pairwise separation problem, achieving 99.39% accuracy. Second, we utilize those findings to formulate a generalized classifier for fake content, by analyzing proposed signal transformations and corresponding feature sets. Third, we generate novel signal maps and employ a CNN to improve our traditional classifier for detecting synthetic content. Lastly, we release an "in the wild" dataset of fake portrait videos that we collected as a part of our evaluation process. We evaluate FakeCatcher on several datasets, resulting with 96%, 94.65%, 91.50%, and 91.07% accuracies, on Face Forensics, Face Forensics++, CelebDF, and on our new Deep Fakes Dataset respectively. We also analyze signals from various facial regions, under image distortions, with varying segment durations, from different generators, against unseen datasets, and under several dimensionality reduction techniques.

arXiv commentTo appear in IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), accepted July 2020. Dataset: http://bit.ly/FakeCatcher

References in corpus (13)

• Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• MesoNet: a Compact Facial Video Forgery Detection Network
• Exposing DeepFake Videos By Detecting Face Warping Artifacts
• DeepFakes: a New Threat to Face Recognition? Assessment and Detection
• FaceForensics: A Large-scale Video Dataset for Forgery Detection in Human Faces
• Identification of Deep Network Generated Images Using Disparities in Color Components
• Neural Photo Editing with Introspective Adversarial Networks
• Aligned and Non-Aligned Double JPEG Detection Using Convolutional Neural Networks
• In Ictu Oculi: Exposing AI Generated Fake Face Videos by Detecting Eye Blinking
• ExprGAN: Facial Expression Editing with Controllable Expression Intensity
• GANimation: Anatomically-aware Facial Animation from a Single Image
• Face Transfer with Generative Adversarial Network

Cited by in corpus (31)

• Media Forensics and DeepFakes: an overview
• Deep Learning for Deepfakes Creation and Detection: A Survey
• Deepfake Video Detection Using Convolutional Vision Transformer
• Deepfake Detection: A Comprehensive Survey from the Reliability Perspective
• FedForgery: Generalized Face Forgery Detection with Residual Federated Learning
• A Review of Deep Learning-based Approaches for Deepfake Content Detection
• Contrast-Phys+: Unsupervised and Weakly-supervised Video-based Remote Physiological Measurement via Spatiotemporal Contrast
• DeepFakesON-Phys: DeepFakes Detection based on Heart Rate Estimation
• Where Do Deep Fakes Look? Synthetic Face Detection via Gaze Tracking
• Deepfake: Definitions, Performance Metrics and Standards, Datasets and Benchmarks, and a Meta-Review
• FFR_FD: Effective and Fast Detection of DeepFakes Based on Feature Point Defects
• WildDeepfake: A Challenging Real-World Dataset for Deepfake Detection
• Learning to Discover Forgery Cues for Face Forgery Detection
• Multi-attentional Deepfake Detection
• Adversarially robust deepfake media detection using fused convolutional neural network predictions
• Deep Learning for Face Anti-Spoofing: A Survey
• How Do the Hearts of Deep Fakes Beat? Deep Fake Source Detection via Interpreting Residuals with Biological Signals
• FakeTagger: Robust Safeguards against DeepFake Dissemination via Provenance Tracking
• A Survey On Anti-Spoofing Methods For Face Recognition with RGB Cameras of Generic Consumer Devices
• Fake face detection via adaptive manipulation traces extraction network
• DeepFN: Towards Generalizable Facial Action Unit Recognition with Deep Face Normalization
• Defending Your Voice: Adversarial Attack on Voice Conversion
• PDIWS: Thermal Imaging Dataset for Person Detection in Intrusion Warning Systems
• Detection of Deepfake Videos Using Long Distance Attention
• Video Transformer for Deepfake Detection with Incremental Learning
• Introducing Explicit Gaze Constraints to Face Swapping
• Exposing Deepfake with Pixel-wise AR and PPG Correlation from Faint Signals
• Generative Models for Security: Attacks, Defenses, and Opportunities
• DeepFake Detection with Inconsistent Head Poses: Reproducibility and Analysis
• Tell-Tale Watermarks for Explanatory Reasoning in Synthetic Media Forensics
• Modeling the Nonsmoothness of Modern Neural Networks