Contrastive Learning for Lifted Networks
arXiv:1905.02507
Abstract
In this work we address supervised learning of neural networks via lifted network formulations. Lifted networks are interesting because they allow training on massively parallel hardware and assign energy models to discriminatively trained neural networks. We demonstrate that the training methods for lifted networks proposed in the literature have significant limitations and show how to use a contrastive loss to address those limitations. We demonstrate that this contrastive training approximates back-propagation in theory and in practice and that it is superior to the training objective regularly used for lifted networks.
arXiv comment9 pages, BMVC 2019
References in corpus (6)
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
- Training Neural Networks Without Gradients: A Scalable ADMM Approach
- Distributed optimization of deeply nested systems
- Fenchel Lifted Networks: A Lagrange Relaxation of Neural Network Training
- Lifted Neural Networks
- Trusting SVM for Piecewise Linear CNNs