Understanding the Role of Adversarial Regularization in Supervised Learning

Abstract

Despite numerous attempts sought to provide empirical evidence of adversarial regularization outperforming sole supervision, the theoretical understanding of such phenomena remains elusive. In this study, we aim to resolve whether adversarial regularization indeed performs better than sole supervision at a fundamental level. To bring this insight into fruition, we study vanishing gradient issue, asymptotic iteration complexity, gradient flow and provable convergence in the context of sole supervision and adversarial regularization. The key ingredient is a theoretical justification supported by empirical evidence of adversarial acceleration in gradient descent. In addition, motivated by a recently introduced unit-wise capacity based generalization bound, we analyze the generalization error in adversarial framework. Guided by our observation, we cast doubts on the ability of this measure to explain generalization. We therefore leave as open questions to explore new measures that can explain generalization behavior in adversarial learning. Furthermore, we observe an intriguing phenomenon in the neural embedded vector space while contrasting adversarial learning with sole supervision.

arXiv commentUnder Review

References in corpus (12)

• Conditional Generative Adversarial Nets
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• ADADELTA: An Adaptive Learning Rate Method
• Super-resolution of Sentinel-2 images: Learning a globally applicable deep neural network
• Learning to Generate Images of Outdoor Scenes from Attributes and Semantic Layouts
• Why gradient clipping accelerates training: A theoretical justification for adaptivity
• On the Convergence of Adaptive Gradient Methods for Nonconvex Optimization
• Model-Predictive Policy Learning with Uncertainty Regularization for Driving in Dense Traffic
• Stochastic Nested Variance Reduction for Nonconvex Optimization
• Generalization in Deep Networks: The Role of Distance from Initialization
• Near-Optimal Algorithms for Minimax Optimization
• Aerial Spectral Super-Resolution using Conditional Adversarial Networks