Rethinking Softmax Cross-Entropy Loss for Adversarial Robustness

Abstract

Previous work shows that adversarially robust generalization requires larger sample complexity, and the same dataset, e.g., CIFAR-10, which enables good standard accuracy may not suffice to train robust models. Since collecting new training data could be costly, we focus on better utilizing the given data by inducing the regions with high sample density in the feature space, which could lead to locally sufficient samples for robust learning. We first formally show that the softmax cross-entropy (SCE) loss and its variants convey inappropriate supervisory signals, which encourage the learned feature points to spread over the space sparsely in training. This inspires us to propose the Max-Mahalanobis center (MMC) loss to explicitly induce dense feature regions in order to benefit robustness. Namely, the MMC loss encourages the model to concentrate on learning ordered and compact representations, which gather around the preset optimal centers for different classes. We empirically demonstrate that applying the MMC loss can significantly improve robustness even under strong adaptive attacks, while keeping state-of-the-art accuracy on clean inputs with little extra computation compared to the SCE loss.

ICLR 2020

References in corpus (11)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Deep Learning Face Representation by Joint Identification-Verification
• ZOO: Zeroth Order Optimization based Black-box Attacks to Deep Neural Networks without Training Substitute Models
• Theoretically Principled Trade-off between Robustness and Accuracy
• On Evaluating Adversarial Robustness
• Improving Adversarial Robustness via Promoting Ensemble Diversity
• Are Labels Required for Improving Adversarial Robustness?
• Adversarial Examples Are a Natural Consequence of Test Error in Noise
• Robustness of classifiers: from adversarial to random noise
• Training for Faster Adversarial Robustness Verification via Inducing ReLU Stability
• Face Recognition via Centralized Coordinate Learning

Cited by in corpus (27)

• Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks
• On Adaptive Attacks to Adversarial Example Defenses
• Minimally distorted Adversarial Examples with a Fast Adaptive Boundary Attack
• RobustBench: a standardized adversarial robustness benchmark
• Boosting Adversarial Training with Hypersphere Embedding
• Inspect, Understand, Overcome: A Survey of Practical Methods for AI Safety
• MiCE: Mixture of Contrastive Experts for Unsupervised Image Clustering
• Denoised Internal Models: a Brain-Inspired Autoencoder against Adversarial Attacks
• RNAS-CL: Robust Neural Architecture Search by Cross-Layer Knowledge Distillation
• DDPNOpt: Differential Dynamic Programming Neural Optimizer
• Improve Generalization and Robustness of Neural Networks via Weight Scale Shifting Invariant Regularizations
• Improve Adversarial Robustness via Weight Penalization on Classification Layer
• Picket: Guarding Against Corrupted Data in Tabular Data during Learning and Inference
• D-square-B: Deep Distribution Bound for Natural-looking Adversarial Attack
• Enhance Diffusion to Improve Robust Generalization
• Enhanced countering adversarial attacks via input denoising and feature restoring
• Adversarial robustness via stochastic regularization of neural activation sensitivity
• The art of defense: letting networks fool the attacker
• Can audio-visual integration strengthen robustness under multimodal attacks?
• Generative Max-Mahalanobis Classifiers for Image Classification, Generation and More
• Introducing the DOME Activation Functions
• Improving Adversarial Robustness for Free with Snapshot Ensemble
• Efficient Adversarial Input Generation via Neural Net Patching
• Adversarially Robust and Explainable Model Compression with On-Device Personalization for Text Classification
• Likelihood Landscapes: A Unifying Principle Behind Many Adversarial Defenses
• Training Provably Robust Models by Polyhedral Envelope Regularization
• Mixing between the Cross Entropy and the Expectation Loss Terms