LTD: Low Temperature Distillation for Gradient Masking-free Adversarial Training

Abstract

Adversarial training is a widely adopted strategy to bolster the robustness of neural network models against adversarial attacks. This paper revisits the fundamental assumptions underlying image classification and suggests that representing data as one-hot labels is a key factor that leads to vulnerabilities. However, in real-world datasets, data ambiguity often arises, with samples exhibiting characteristics of multiple classes, rendering one-hot label representations imprecise. To address this, we introduce a novel approach, Low-Temperature Distillation (LTD), designed to refine label representations. Unlike previous approaches, LTD incorporates a relatively low temperature in the teacher model, while maintaining a fixed temperature for the student model during both training and inference. This strategy not only refines assumptions about data distribution but also strengthens model robustness and avoids the gradient masking problem commonly encountered in defensive distillation. Experimental results demonstrate the efficacy of the proposed method when combined with existing frameworks, achieving robust accuracy rates of 58.19%, 31.13%, and 42.08% on the CIFAR-10, CIFAR-100, and ImageNet datasets, respectively, without the need for additional data.

References in corpus (19)

• Distilling the Knowledge in a Neural Network
• YOLOv4: Optimal Speed and Accuracy of Object Detection
• Knowledge Distillation: A Survey
• Wide Residual Networks
• A Survey of Deep Learning Techniques for Autonomous Driving
• ZOO: Zeroth Order Optimization based Black-box Attacks to Deep Neural Networks without Training Substitute Models
• Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning
• Theoretically Principled Trade-off between Robustness and Accuracy
• When Does Label Smoothing Help?
• Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks
• On Evaluating Adversarial Robustness
• Fast is better than free: Revisiting adversarial training
• Adversarial Risk and the Dangers of Evaluating Against Weak Attacks
• Natural Adversarial Examples
• Uncovering the Limits of Adversarial Training against Norm-Bounded Adversarial Examples
• RobustBench: a standardized adversarial robustness benchmark
• Unrestricted Adversarial Examples
• Four Things Everyone Should Know to Improve Batch Normalization
• Learnable Boundary Guided Adversarial Training

Cited by in corpus (2)

• How Deep Learning Sees the World: A Survey on Adversarial Attacks & Defenses
• Adversarial Robustness through the Lens of Convolutional Filters