Benchmarking Adversarial Robustness

Abstract

Deep neural networks are vulnerable to adversarial examples, which becomes one of the most important research problems in the development of deep learning. While a lot of efforts have been made in recent years, it is of great significance to perform correct and complete evaluations of the adversarial attack and defense algorithms. In this paper, we establish a comprehensive, rigorous, and coherent benchmark to evaluate adversarial robustness on image classification tasks. After briefly reviewing plenty of representative attack and defense methods, we perform large-scale experiments with two robustness curves as the fair-minded evaluation criteria to fully understand the performance of these methods. Based on the evaluation results, we draw several important findings and provide insights for future research.

References in corpus (9)

• Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples
• On Evaluating Adversarial Robustness
• Technical Report on the CleverHans v2.1.0 Adversarial Examples Library
• A study of the effect of JPG compression on adversarial images
• Improving Adversarial Robustness via Promoting Ensemble Diversity
• Semidefinite relaxations for certifying robustness to adversarial examples
• Adv-BNN: Improved Adversarial Defense through Robust Bayesian Neural Network
• Practical Black-Box Attacks against Machine Learning
• A critique of the DeepSec Platform for Security Analysis of Deep Learning Models

Cited by in corpus (11)

• Boosting Adversarial Training with Hypersphere Embedding
• Adversarial Attacks and Defenses for Speech Recognition Systems
• Estimating the Brittleness of AI: Safety Integrity Levels and the Need for Testing Out-Of-Distribution Performance
• Model-Based Robust Deep Learning: Generalizing to Natural, Out-of-Distribution Data
• Boosting Transferability of Targeted Adversarial Examples via Hierarchical Generative Networks
• Medical Aegis: Robust adversarial protectors for medical images
• RAIN: A Simple Approach for Robust and Accurate Image Classification Networks
• Noise Optimization for Artificial Neural Networks
• A Hierarchical Feature Constraint to Camouflage Medical Adversarial Attacks
• Dynamic Defense Approach for Adversarial Robustness in Deep Neural Networks via Stochastic Ensemble Smoothed Model
• Accumulative Poisoning Attacks on Real-time Data