RobustART: Benchmarking Robustness on Architecture Design and Training Techniques
arXiv:2109.05211
Abstract
Deep neural networks (DNNs) are vulnerable to adversarial noises, which motivates the benchmark of model robustness. Existing benchmarks mainly focus on evaluating defenses, but there are no comprehensive studies of how architecture design and training techniques affect robustness. Comprehensively benchmarking their relationships is beneficial for better understanding and developing robust DNNs. Thus, we propose RobustART, the first comprehensive Robustness investigation benchmark on ImageNet regarding ARchitecture design (49 human-designed off-the-shelf architectures and 1200+ networks from neural architecture search) and Training techniques (10+ techniques, e.g., data augmentation) towards diverse noises (adversarial, natural, and system noises). Extensive experiments substantiated several insights for the first time, e.g., (1) adversarial training is effective for the robustness against all noises types for Transformers and MLP-Mixers; (2) given comparable model sizes and aligned training settings, CNNs > Transformers > MLP-Mixers on robustness against natural and system noises; Transformers > MLP-Mixers > CNNs on adversarial robustness; (3) for some light-weight architectures, increasing model sizes or using extra data cannot improve robustness. Our benchmark presents: (1) an open-source platform for comprehensive robustness evaluation; (2) a variety of pre-trained models to facilitate robustness evaluation; and (3) a new view to better understand the mechanism towards designing robust DNNs. We will continuously develop to this ecosystem for the community.
References in corpus (20)
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Distilling the Knowledge in a Neural Network
- Explaining and Harnessing Adversarial Examples
- On Calibration of Modern Neural Networks
- MLP-Mixer: An all-MLP Architecture for Vision
- Theoretically Principled Trade-off between Robustness and Accuracy
- DeepID3: Face Recognition with Very Deep Neural Networks
- Delving into Transferable Adversarial Examples and Black-box Attacks
- On Evaluating Adversarial Robustness
- Regularizing Neural Networks by Penalizing Confident Output Distributions
- Intriguing Properties of Vision Transformers
- Skip Connections Matter: On the Transferability of Adversarial Examples Generated with ResNets
- Partial success in closing the gap between human and machine vision
- AtomNAS: Fine-Grained End-to-End Neural Architecture Search
- Label Smoothing and Logit Squeezing: A Replacement for Adversarial Training?
- Stochastic Weight Averaging in Parallel: Large-Batch Training that Generalizes Well
- Label Smoothing and Adversarial Robustness
- Yet Another Intermediate-Level Attack
- How do SGD hyperparameters in natural training affect adversarial robustness?
- Real World Robustness from Systematic Noise