Lipschitz Bounded Equilibrium Networks

Abstract

This paper introduces new parameterizations of equilibrium neural networks, i.e. networks defined by implicit equations. This model class includes standard multilayer and residual networks as special cases. The new parameterization admits a Lipschitz bound during training via unconstrained optimization: no projections or barrier functions are required. Lipschitz bounds are a common proxy for robustness and appear in many generalization bounds. Furthermore, compared to previous works we show well-posedness (existence of solutions) under less restrictive conditions on the network weights and more natural assumptions on the activation functions: that they are monotone and slope restricted. These results are proved by establishing novel connections with convex optimization, operator splitting on non-Euclidean spaces, and contracting neural ODEs. In image classification experiments we show that the Lipschitz bounds are very accurate and improve robustness to adversarial attacks.

Conference submission, 19 pages

References in corpus (4)

• Evaluating Robustness of Neural Networks with Mixed Integer Programming
• Deep Equilibrium Models
• Monotone operator equilibrium networks
• An Analysis of the Expressiveness of Deep Neural Network Architectures Based on Their Lipschitz Constants

Cited by in corpus (8)

• Contraction Theory for Nonlinear Stability Analysis and Learning-based Control: A Tutorial Overview
• Perspectives on Contractivity in Control, Optimization, and Learning
• Offset-free setpoint tracking using neural network controllers
• Abstract Interpretation of Fixpoint Iterators with Applications to Neural Networks
• Learning Stable and Robust Linear Parameter-Varying State-Space Models
• Optimization Induced Equilibrium Networks
• From Contraction Theory to Fixed Point Algorithms on Riemannian and Non-Euclidean Spaces
• Robust Implicit Networks via Non-Euclidean Contractions