Fenchel Lifted Networks: A Lagrange Relaxation of Neural Network Training

Abstract

Despite the recent successes of deep neural networks, the corresponding training problem remains highly non-convex and difficult to optimize. Classes of models have been proposed that introduce greater structure to the objective function at the cost of lifting the dimension of the problem. However, these lifted methods sometimes perform poorly compared to traditional neural networks. In this paper, we introduce a new class of lifted models, Fenchel lifted networks, that enjoy the same benefits as previous lifted models, without suffering a degradation in performance over classical networks. Our model represents activation functions as equivalent biconvex constraints and uses Lagrange Multipliers to arrive at a rigorous lower bound of the traditional neural network training problem. This model is efficiently trained using block-coordinate descent and is parallelizable across data points and/or layers. We compare our model against standard fully connected and convolutional networks and show that we are able to match or beat their performance.

References in corpus (5)

• Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
• Training Neural Networks Without Gradients: A Scalable ADMM Approach
• Distributed optimization of deeply nested systems
• A Proximal Block Coordinate Descent Algorithm for Deep Neural Network Training
• Lifted Neural Networks

Cited by in corpus (6)

• You Only Propagate Once: Accelerating Adversarial Training via Maximal Principle
• On Training Implicit Models
• Neural Network Training as an Optimal Control Problem: An Augmented Lagrangian Approach
• LocoProp: Enhancing BackProp via Local Loss Optimization
• Stochastic Block-ADMM for Training Deep Networks
• Contrastive Learning for Lifted Networks