Fast and Provable ADMM for Learning with Generative Priors

Abstract

In this work, we propose a (linearized) Alternating Direction Method-of-Multipliers (ADMM) algorithm for minimizing a convex function subject to a nonconvex constraint. We focus on the special case where such constraint arises from the specification that a variable should lie in the range of a neural network. This is motivated by recent successful applications of Generative Adversarial Networks (GANs) in tasks like compressive sensing, denoising and robustness against adversarial examples. The derived rates for our algorithm are characterized in terms of certain geometric properties of the generator network, which we show hold for feedforward architectures, under mild assumptions. Unlike gradient descent (GD), it can efficiently handle non-smooth objectives as well as exploit efficient partial minimization procedures, thus being faster in many practical scenarios.

References in corpus (12)

• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
• Empirical Evaluation of Rectified Activations in Convolutional Network
• Which Training Methods for GANs do actually Converge?
• The Robust Manifold Defense: Adversarial Training using Generative Models
• Adversarial Audio Synthesis
• Correction by Projection: Denoising Images with Generative Adversarial Networks
• Linearized ADMM for Non-convex Non-smooth Optimization with Convergence Analysis
• Finding Mixed Nash Equilibria of Generative Adversarial Networks

Cited by in corpus (4)

• Solving Inverse Problems by Joint Posterior Maximization with Autoencoding Prior
• Solving Inverse Problems with Hybrid Deep Image Priors: the challenge of preventing overfitting
• Distributed and Scalable Optimization for Robust Proton Treatment Planning
• Inverse problem regularization with hierarchical variational autoencoders