Orthogonalizing Convolutional Layers with the Cayley Transform
arXiv:2104.07167
Abstract
Recent work has highlighted several advantages of enforcing orthogonality in the weight layers of deep networks, such as maintaining the stability of activations, preserving gradient norms, and enhancing adversarial robustness by enforcing low Lipschitz constants. Although numerous methods exist for enforcing the orthogonality of fully-connected layers, those for convolutional layers are more heuristic in nature, often focusing on penalty methods or limited classes of convolutions. In this work, we propose and evaluate an alternative approach to directly parameterize convolutional layers that are constrained to be orthogonal. Specifically, we propose to apply the Cayley transform to a skew-symmetric convolution in the Fourier domain, so that the inverse convolution needed by the Cayley transform can be computed efficiently. We compare our method to previous Lipschitz-constrained and orthogonal convolutional layers and show that it indeed preserves orthogonality to a high degree even for large convolutions. Applied to the problem of certified adversarial robustness, we show that networks incorporating the layer outperform existing deterministic methods for certified defense against -norm-bounded adversaries, while scaling to larger architectures than previously investigated. Code is available at https://github.com/locuslab/orthogonal-convolutions.
To appear in ICLR 2021
References in corpus (15)
- Wide Residual Networks
- Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
- Provable defenses against adversarial examples via the convex outer adversarial polytope
- Certified Adversarial Robustness via Randomized Smoothing
- An approach to reachability analysis for feed-forward ReLU neural networks
- Evaluating the Robustness of Neural Networks: An Extreme Value Theory Approach
- On orthogonality and learning recurrent networks with long term dependencies
- Fixup Initialization: Residual Learning Without Normalization
- Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Group
- Regularizing CNNs with Locally Constrained Decorrelations
- Emerging Convolutions for Generative Normalizing Flows
- Provable Benefit of Orthogonal Initialization in Optimizing Deep Linear Networks
- Preventing Gradient Attenuation in Lipschitz Constrained Convolutional Networks
- Efficient Riemannian Optimization on the Stiefel Manifold via the Cayley Transform
- Remarks on the Cayley Representation of Orthogonal Matrices and on Perturbing the Diagonal of a Matrix to Make it Invertible