Reverse Engineering of Generative Models: Inferring Model Hyperparameters from Generated Images

Abstract

State-of-the-art (SOTA) Generative Models (GMs) can synthesize photo-realistic images that are hard for humans to distinguish from genuine photos. Identifying and understanding manipulated media are crucial to mitigate the social concerns on the potential misuse of GMs. We propose to perform reverse engineering of GMs to infer model hyperparameters from the images generated by these models. We define a novel problem, ``model parsing", as estimating GM network architectures and training loss functions by examining their generated images -- a task seemingly impossible for human beings. To tackle this problem, we propose a framework with two components: a Fingerprint Estimation Network (FEN), which estimates a GM fingerprint from a generated image by training with four constraints to encourage the fingerprint to have desired properties, and a Parsing Network (PN), which predicts network architecture and loss functions from the estimated fingerprints. To evaluate our approach, we collect a fake image dataset with $100$K images generated by $116$ different GMs. Extensive experiments show encouraging results in parsing the hyperparameters of the unseen models. Finally, our fingerprint estimation can be leveraged for deepfake detection and image attribution, as we show by reporting SOTA results on both the deepfake detection (Celeb-DF) and image attribution benchmarks.

Accepted at Transactions on Pattern Analysis and Machine Intelligence

References in corpus (15)

• Auto-Encoding Variational Bayes
• Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising
• Spectral Normalization for Generative Adversarial Networks
• Self-Attention Generative Adversarial Networks
• MesoNet: a Compact Facial Video Forgery Detection Network
• LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
• BEGAN: Boundary Equilibrium Generative Adversarial Networks
• Exposing DeepFake Videos By Detecting Face Warping Artifacts
• Disentangling by Factorising
• MMD GAN: Towards Deeper Understanding of Moment Matching Network
• Understanding disentangling in $\beta$-VAE
• The Cramer Distance as a Solution to Biased Wasserstein Gradients
• MAGAN: Margin Adaptation for Generative Adversarial Networks
• DeepFake Detection Based on the Discrepancy Between the Face and its Context
• Feature Importance Estimation with Self-Attention Networks