Lifted Neural Networks

Abstract

We describe a novel family of models of multi- layer feedforward neural networks in which the activation functions are encoded via penalties in the training problem. Our approach is based on representing a non-decreasing activation function as the argmin of an appropriate convex optimiza- tion problem. The new framework allows for algo- rithms such as block-coordinate descent methods to be applied, in which each step is composed of a simple (no hidden layer) supervised learning problem that is parallelizable across data points and/or layers. Experiments indicate that the pro- posed models provide excellent initial guesses for weights for standard neural networks. In addi- tion, the model provides avenues for interesting extensions, such as robustness against noisy in- puts and optimizing over parameters in activation functions.

References in corpus (5)

• The Marginal Value of Adaptive Gradient Methods in Machine Learning
• Distributed optimization of deeply nested systems
• PCA-Initialized Deep Neural Networks Applied To Document Image Analysis
• Iterative Hessian sketch: Fast and accurate solution approximation for constrained least-squares
• Trusting SVM for Piecewise Linear CNNs

Cited by in corpus (6)

• You Only Propagate Once: Accelerating Adversarial Training via Maximal Principle
• Fenchel Lifted Networks: A Lagrange Relaxation of Neural Network Training
• LocoProp: Enhancing BackProp via Local Loss Optimization
• Stochastic Block-ADMM for Training Deep Networks
• Contrastive Learning for Lifted Networks
• Inertial Proximal Deep Learning Alternating Minimization for Efficient Neutral Network Training