Parseval Networks: Improving Robustness to Adversarial Examples

Abstract

We introduce Parseval networks, a form of deep neural networks in which the Lipschitz constant of linear, convolutional and aggregation layers is constrained to be smaller than 1. Parseval networks are empirically and theoretically motivated by an analysis of the robustness of the predictions made by deep neural networks when their input is subject to an adversarial perturbation. The most important feature of Parseval networks is to maintain weight matrices of linear and convolutional layers to be (approximately) Parseval tight frames, which are extensions of orthogonal matrices to non-square matrices. We describe how these constraints can be maintained efficiently during SGD. We show that Parseval networks match the state-of-the-art in terms of accuracy on CIFAR-10/100 and Street View House Numbers (SVHN) while being more robust than their vanilla counterpart against adversarial examples. Incidentally, Parseval networks also tend to train faster and make a better usage of the full capacity of the networks.

arXiv commentsubmitted

Cited by in corpus (67)

• DeepXplore: Automated Whitebox Testing of Deep Learning Systems
• Spatial-Temporal Graph ODE Networks for Traffic Flow Forecasting
• Countering Adversarial Images using Input Transformations
• Understanding Adversarial Training: Increasing Local Stability of Neural Nets through Robust Optimization
• Prevalence of Neural Collapse during the terminal phase of deep learning training
• Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learning
• PixelDefend: Leveraging Generative Models to Understand and Defend against Adversarial Examples
• Word Translation Without Parallel Data
• DeepTest: Automated Testing of Deep-Neural-Network-driven Autonomous Cars
• Measuring the tendency of CNNs to Learn Surface Statistical Regularities
• Deceiving End-to-End Deep Learning Malware Detectors using Adversarial Examples
• Interpretation of Neural Networks is Fragile
• Towards Practical Verification of Machine Learning: The Case of Computer Vision Systems
• i-RevNet: Deep Invertible Networks
• Adversarial Explanations for Understanding Image Classification Decisions and Improved Neural Network Robustness
• Defending against Adversarial Images using Basis Functions Transformations
• Detecting Adversarial Samples for Deep Neural Networks through Mutation Testing
• Continuous Graph Neural Networks
• NAIS-Net: Stable Deep Networks from Non-Autonomous Differential Equations
• Theoretical evidence for adversarial robustness through randomization
• Daedalus: Breaking Non-Maximum Suppression in Object Detection via Adversarial Examples
• Exactly Computing the Local Lipschitz Constant of ReLU Networks
• Adversarial Learning in Statistical Classification: A Comprehensive Review of Defenses Against Attacks
• Intriguing Properties of Adversarial Examples
• There is Limited Correlation between Coverage and Robustness for Deep Neural Networks
• Stabilizing Gradients for Deep Neural Networks via Efficient SVD Parameterization
• Lipschitzness Is All You Need To Tame Off-policy Generative Adversarial Imitation Learning
• ConvNets and ImageNet Beyond Accuracy: Understanding Mistakes and Uncovering Biases
• Adversarially Robust Training through Structured Gradient Regularization
• Large Margin Deep Networks for Classification
• Improving Network Robustness against Adversarial Attacks with Compact Convolution
• Spectral Normalisation for Deep Reinforcement Learning: an Optimisation Perspective
• Layer-wise Characterization of Latent Information Leakage in Federated Learning
• Adversarial attacks hidden in plain sight
• Adversarial Extreme Multi-label Classification
• Retrieval-Augmented Convolutional Neural Networks for Improved Robustness against Adversarial Examples
• Adversarial Image Translation: Unrestricted Adversarial Examples in Face Recognition Systems
• Learning Unsupervised Word Mapping by Maximizing Mean Discrepancy
• Some open questions on morphological operators and representations in the deep learning era
• Analysis of Invariance and Robustness via Invertibility of ReLU-Networks
• Information Geometry of Orthogonal Initializations and Training
• Fibres of Failure: Classifying errors in predictive processes
• RNAS-CL: Robust Neural Architecture Search by Cross-Layer Knowledge Distillation
• Training Certifiably Robust Neural Networks with Efficient Local Lipschitz Bounds
• A Dynamical View on Optimization Algorithms of Overparameterized Neural Networks
• Structure-Preserving Transformation: Generating Diverse and Transferable Adversarial Examples
• Predicting Adversarial Examples with High Confidence
• Hardware Accelerator for Adversarial Attacks on Deep Learning Neural Networks
• Learning Diverse-Structured Networks for Adversarial Robustness
• All your loss are belong to Bayes
• What Deep CNNs Benefit from Global Covariance Pooling: An Optimization Perspective
• Lifted Regression/Reconstruction Networks
• Noise Optimization for Artificial Neural Networks
• MAT: A Multi-strength Adversarial Training Method to Mitigate Adversarial Attacks
• Regularized Ensembles and Transferability in Adversarial Learning
• Efficient Proximal Mapping of the 1-path-norm of Shallow Networks
• A Hierarchical Feature Constraint to Camouflage Medical Adversarial Attacks
• Notes on Margin Training and Margin p-Values for Deep Neural Network Classifiers
• Local Linearity and Double Descent in Catastrophic Overfitting
• NeuralDP Differentially private neural networks by design
• Regularized Training and Tight Certification for Randomized Smoothed Classifier with Provable Robustness
• Optimization on Product Submanifolds of Convolution Kernels
• Defense-guided Transferable Adversarial Attacks
• DCA: Diversified Co-Attention towards Informative Live Video Commenting
• Sparse Coding Frontend for Robust Neural Networks
• Cross-Lingual Low-Resource Set-to-Description Retrieval for Global E-Commerce
• On Connections between Regularizations for Improving DNN Robustness