ResNets Ensemble via the Feynman-Kac Formalism to Improve Natural and Robust Accuracies

Abstract

Empirical adversarial risk minimization (EARM) is a widely used mathematical framework to robustly train deep neural nets (DNNs) that are resistant to adversarial attacks. However, both natural and robust accuracies, in classifying clean and adversarial images, respectively, of the trained robust models are far from satisfactory. In this work, we unify the theory of optimal control of transport equations with the practice of training and testing of ResNets. Based on this unified viewpoint, we propose a simple yet effective ResNets ensemble algorithm to boost the accuracy of the robustly trained model on both clean and adversarial images. The proposed algorithm consists of two components: First, we modify the base ResNets by injecting a variance specified Gaussian noise to the output of each residual mapping. Second, we average over the production of multiple jointly trained modified ResNets to get the final prediction. These two steps give an approximation to the Feynman-Kac formula for representing the solution of a transport equation with viscosity, or a convection-diffusion equation. For the CIFAR10 benchmark, this simple algorithm leads to a robust model with a natural accuracy of {\bf 85.62}\% on clean images and a robust accuracy of $57.94\%$ under the 20 iterations of the IFGSM attack, which outperforms the current state-of-the-art in defending against IFGSM attack on the CIFAR10. Both natural and robust accuracies of the proposed ResNets ensemble can be improved dynamically as the building block ResNet advances. The code is available at: \url{https://github.com/BaoWangMath/EnResNet}.

arXiv comment18 pages, 6 figures

References in corpus (21)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Transferability in Machine Learning: from Phenomena to Black-Box Attacks using Adversarial Samples
• Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning
• Theoretically Principled Trade-off between Robustness and Accuracy
• Delving into Transferable Adversarial Examples and Black-box Attacks
• Provable defenses against adversarial examples via the convex outer adversarial polytope
• Certified Adversarial Robustness via Randomized Smoothing
• Certified Defenses against Adversarial Examples
• Adversarial Perturbations Against Deep Neural Networks for Malware Classification
• Improving the Adversarial Robustness and Interpretability of Deep Neural Networks by Regularizing their Input Gradients
• Semidefinite relaxations for certifying robustness to adversarial examples
• A Convex Relaxation Barrier to Tight Robustness Verification of Neural Networks
• Rademacher Complexity for Adversarially Robust Generalization
• Ensemble Methods as a Defense to Adversarial Perturbations Against Deep Neural Networks
• Convolutional Neural Networks combined with Runge-Kutta Methods
• Deep Residual Learning and PDEs on Manifold
• Deep Neural Nets with Interpolating Function as Output Activation
• Stochastic Training of Residual Networks: a Differential Equation Viewpoint
• Adversarial Defense via Data Dependent Activation Function and Total Variation Minimization
• Mathematical Analysis of Adversarial Attacks

Cited by in corpus (8)

• A Review on Deep Learning in Medical Image Reconstruction
• Enhancing Discrete Choice Models with Representation Learning
• You Only Propagate Once: Accelerating Adversarial Training via Maximal Principle
• On Robustness of Neural Ordinary Differential Equations
• Achieving Adversarial Robustness via Sparsity
• InfoCNF: An Efficient Conditional Continuous Normalizing Flow with Adaptive Solvers
• Graph Interpolating Activation Improves Both Natural and Robust Accuracies in Data-Efficient Deep Learning
• Towards Optimal Randomized Strategies in Adversarial Example Game