Defensive Quantization: When Efficiency Meets Robustness

Abstract

Neural network quantization is becoming an industry standard to efficiently deploy deep learning models on hardware platforms, such as CPU, GPU, TPU, and FPGAs. However, we observe that the conventional quantization approaches are vulnerable to adversarial attacks. This paper aims to raise people's awareness about the security of the quantized models, and we designed a novel quantization methodology to jointly optimize the efficiency and robustness of deep learning models. We first conduct an empirical study to show that vanilla quantization suffers more from adversarial attacks. We observe that the inferior robustness comes from the error amplification effect, where the quantization operation further enlarges the distance caused by amplified noise. Then we propose a novel Defensive Quantization (DQ) method by controlling the Lipschitz constant of the network during quantization, such that the magnitude of the adversarial noise remains non-expansive during inference. Extensive experiments on CIFAR-10 and SVHN datasets demonstrate that our new quantization method can defend neural networks against adversarial examples, and even achieves superior robustness than their full-precision counterparts while maintaining the same hardware efficiency as vanilla quantization approaches. As a by-product, DQ can also improve the accuracy of quantized models without adversarial attack.

References in corpus (18)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Explaining and Harnessing Adversarial Examples
• Ensemble Adversarial Training: Attacks and Defenses
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networks
• Wide Residual Networks
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Towards Deep Learning Models Resistant to Adversarial Attacks
• Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples
• Trained Ternary Quantization
• PixelDefend: Leveraging Generative Models to Understand and Defend against Adversarial Examples
• Stochastic Activation Pruning for Robust Adversarial Defense
• Defend Deep Neural Networks Against Adversarial Examples via Fixed and Dynamic Quantized Activation Functions
• L2-Nonexpansive Neural Networks
• Attacking Binarized Neural Networks
• Is Robustness the Cost of Accuracy? -- A Comprehensive Study on the Robustness of 18 Deep Image Classification Models

Cited by in corpus (16)

• Bringing AI To Edge: From Deep Learning's Perspective
• Adversarial Machine Learning in Image Classification: A Survey Towards the Defender's Perspective
• When NAS Meets Robustness: In Search of Robust Architectures against Adversarial Attacks
• Generative Low-bitwidth Data Free Quantization
• Noise Sensitivity-Based Energy Efficient and Robust Adversary Detection in Neural Networks
• QUANOS- Adversarial Noise Sensitivity Driven Hybrid Quantization of Neural Networks
• Efficiency-driven Hardware Optimization for Adversarially Robust Neural Networks
• Improving Adversarial Robustness in Weight-quantized Neural Networks
• SoK: How Robust is Image Classification Deep Neural Network Watermarking? (Extended Version)
• Exposing the Robustness and Vulnerability of Hybrid 8T-6T SRAM Memory Architectures to Adversarial Attacks in Deep Neural Networks
• One Model for All Quantization: A Quantized Network Supporting Hot-Swap Bit-Width Adjustment
• Drawing Robust Scratch Tickets: Subnetworks with Inborn Robustness Are Found within Randomly Initialized Networks
• Brain-inspired reverse adversarial examples
• Stochastic-Shield: A Probabilistic Approach Towards Training-Free Adversarial Defense in Quantized CNNs
• DetectX -- Adversarial Input Detection using Current Signatures in Memristive XBar Arrays
• Quantized Neural Networks via {-1, +1} Encoding Decomposition and Acceleration