A spin-glass model for the loss surfaces of generative adversarial networks

Abstract

We present a novel mathematical model that seeks to capture the key design feature of generative adversarial networks (GANs). Our model consists of two interacting spin glasses, and we conduct an extensive theoretical analysis of the complexity of the model's critical points using techniques from Random Matrix Theory. The result is insights into the loss surfaces of large GANs that build upon prior insights for simpler networks, but also reveal new structure unique to this setting.

arXiv comment26 pages, 9 figures

References in corpus (14)

• Conditional Generative Adversarial Nets
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Self-Attention Generative Adversarial Networks
• The Loss Surfaces of Multilayer Networks
• Shaping the learning landscape in neural networks around wide flat minima
• Complex energy landscapes in spiked-tensor and simple glassy models: ruggedness, arrangements of local minima and phase transitions
• Rethinking mean-field glassy dynamics and its relation with the energy landscape: the awkward case of the spherical mixed p-spin model
• Comparing Dynamics: Deep Neural Networks versus Glassy Systems
• Unveiling the structure of wide flat minima in neural networks
• The Full Spectrum of Deepnet Hessians at Scale: Dynamics with SGD Training and Sample Size
• The Loss Surfaces of Neural Networks with General Activation Functions
• The Supersymmetric Method in Random Matrix Theory and Applications to QCD
• Landscape Complexity for the Empirical Risk of Generalized Linear Models
• Beyond Random Matrix Theory for Deep Networks

Cited by in corpus (5)

• Exponential growth of random determinants beyond invariance
• Universal characteristics of deep neural network loss surfaces from random matrix theory
• Local convexity of the TAP free energy and AMP convergence for Z2-synchronization
• Appearance of Random Matrix Theory in Deep Learning
• Landscape Complexity for the Empirical Risk of Generalized Linear Models: Discrimination between Structured Data