Imposing Hard Constraints on Deep Networks: Promises and Limitations

Abstract

Imposing constraints on the output of a Deep Neural Net is one way to improve the quality of its predictions while loosening the requirements for labeled training data. Such constraints are usually imposed as soft constraints by adding new terms to the loss function that is minimized during training. An alternative is to impose them as hard constraints, which has a number of theoretical benefits but has not been explored so far due to the perceived intractability of the problem. In this paper, we show that imposing hard constraints can in fact be done in a computationally feasible way and delivers reasonable results. However, the theoretical benefits do not materialize and the resulting technique is no better than existing ones relying on soft constraints. We analyze the reasons for this and hope to spur other researchers into proposing better solutions.

References in corpus (4)

• Adam: A Method for Stochastic Optimization
• ADADELTA: An Adaptive Learning Rate Method
• MINRES-QLP: a Krylov subspace method for indefinite or singular symmetric systems
• Krylov Subspace Descent for Deep Learning

Cited by in corpus (19)

• Surrogate Modeling for Fluid Flows Based on Physics-Constrained Deep Learning Without Simulation Data
• Constrained-CNN losses for weakly supervised segmentation
• A Semantic Loss Function for Deep Learning with Symbolic Knowledge
• Constrained Deep Networks: Lagrangian Optimization via Log-Barrier Extensions
• Deep Declarative Networks: A New Hope
• Physics informed deep learning for computational elastodynamics without labeled data
• Constrained domain adaptation for Image segmentation
• Analytically Embedding Differential Equation Constraints into Least Squares Support Vector Machines using the Theory of Functional Connections
• Improving Deep Learning Models via Constraint-Based Domain Knowledge: a Brief Survey
• Deep learning calibration of option pricing models: some pitfalls and solutions
• Constrained Deep Learning using Conditional Gradient and Applications in Computer Vision
• Neural Networks with Physics-Informed Architectures and Constraints for Dynamical Systems Modeling
• Convergence of adaptive algorithms for weakly convex constrained optimization
• Inexact Sequential Quadratic Optimization for Minimizing a Stochastic Objective Function Subject to Deterministic Nonlinear Equality Constraints
• Deep Local Volatility
• Distortion-controlled Training for End-to-end Reverberant Speech Separation with Auxiliary Autoencoding Loss
• Fast Jacobian-Vector Product for Deep Networks
• Longitudinal Pooling & Consistency Regularization to Model Disease Progression from MRIs
• One Size Fits All: Can We Train One Denoiser for All Noise Levels?