The Inductive Bias of Restricted f-GANs

Abstract

Generative adversarial networks are a novel method for statistical inference that have achieved much empirical success; however, the factors contributing to this success remain ill-understood. In this work, we attempt to analyze generative adversarial learning -- that is, statistical inference as the result of a game between a generator and a discriminator -- with the view of understanding how it differs from classical statistical inference solutions such as maximum likelihood inference and the method of moments. Specifically, we provide a theoretical characterization of the distribution inferred by a simple form of generative adversarial learning called restricted f-GANs -- where the discriminator is a function in a given function class, the distribution induced by the generator is restricted to lie in a pre-specified distribution class and the objective is similar to a variational form of the f-divergence. A consequence of our result is that for linear KL-GANs -- that is, when the discriminator is a linear function over some feature space and f corresponds to the KL-divergence -- the distribution induced by the optimal generator is neither the maximum likelihood nor the method of moments solution, but an interesting combination of both.

References in corpus (10)

• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Estimating divergence functionals and the likelihood ratio by convex risk minimization
• Gradient descent GAN optimization is locally stable
• Convergence Problems with Generative Adversarial Networks (GANs)
• On the Limitations of First-Order Approximation in GAN Dynamics
• Understanding GANs: the LQG Setting
• Approximability of Discriminators Implies Diversity in GANs
• Nonparametric Density Estimation under Adversarial Losses
• f-GANs in an Information Geometric Nutshell
• Some Theoretical Properties of GANs

Cited by in corpus (8)

• Adversarial-Residual-Coarse-Graining: Applying machine learning theory to systematic molecular coarse-graining
• How Well Generative Adversarial Networks Learn Distributions
• Interaction Matters: A Note on Non-asymptotic Local Convergence of Generative Adversarial Networks
• Capacity Bounded Differential Privacy
• $(f,\Gamma )$-Divergences: Interpolating between $f$-Divergences and Integral Probability Metrics
• Distributional Robustness with IPMs and links to Regularization and GANs
• Adversarial Networks and Autoencoders: The Primal-Dual Relationship and Generalization Bounds
• Bridging the Gap Between $f$-GANs and Wasserstein GANs