An Integrated Approach to Produce Robust Models with High Efficiency

Abstract

Deep Neural Networks (DNNs) needs to be both efficient and robust for practical uses. Quantization and structure simplification are promising ways to adapt DNNs to mobile devices, and adversarial training is the most popular method to make DNNs robust. In this work, we try to obtain both features by applying a convergent relaxation quantization algorithm, Binary-Relax (BR), to a robust adversarial-trained model, ResNets Ensemble via Feynman-Kac Formalism (EnResNet). We also discover that high precision, such as ternary (tnn) and 4-bit, quantization will produce sparse DNNs. However, this sparsity is unstructured under advarsarial training. To solve the problems that adversarial training jeopardizes DNNs' accuracy on clean images and the struture of sparsity, we design a trade-off loss function that helps DNNs preserve their natural accuracy and improve the channel sparsity. With our trade-off loss function, we achieve both goals with no reduction of resistance under weak attacks and very minor reduction of resistance under strong attcks. Together with quantized EnResNet with trade-off loss function, we provide robust models that have high efficiency.

References in corpus (10)

• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples
• Theoretically Principled Trade-off between Robustness and Accuracy
• Countering Adversarial Images using Input Transformations
• Robustness May Be at Odds with Accuracy
• Certifying Some Distributional Robustness with Principled Adversarial Training
• On the Convergence and Robustness of Adversarial Training
• NATTACK: Learning the Distributions of Adversarial Examples for an Improved Black-Box Attack on Deep Neural Networks
• Deep Residual Learning and PDEs on Manifold
• Deep Neural Nets with Interpolating Function as Output Activation