A Winning Hand: Compressing Deep Networks Can Improve Out-Of-Distribution Robustness

Abstract

Successful adoption of deep learning (DL) in the wild requires models to be: (1) compact, (2) accurate, and (3) robust to distributional shifts. Unfortunately, efforts towards simultaneously meeting these requirements have mostly been unsuccessful. This raises an important question: Is the inability to create Compact, Accurate, and Robust Deep neural networks (CARDs) fundamental? To answer this question, we perform a large-scale analysis of popular model compression techniques which uncovers several intriguing patterns. Notably, in contrast to traditional pruning approaches (e.g., fine tuning and gradual magnitude pruning), we find that "lottery ticket-style" approaches can surprisingly be used to produce CARDs, including binary-weight CARDs. Specifically, we are able to create extremely compact CARDs that, compared to their larger counterparts, have similar test accuracy and matching (or better) robustness -- simply by pruning and (optionally) quantizing. Leveraging the compactness of CARDs, we develop a simple domain-adaptive test-time ensembling approach (CARD-Decks) that uses a gating module to dynamically select appropriate CARDs from the CARD-Deck based on their spectral-similarity with test samples. The proposed approach builds a "winning hand'' of CARDs that establishes a new state-of-the-art (on RobustBench) on CIFAR-10-C accuracies (i.e., 96.8% standard and 92.75% robust) and CIFAR-100-C accuracies (80.6% standard and 71.3% robust) with better memory usage than non-compressed baselines (pretrained CARDs and CARD-Decks available at https://github.com/RobustBench/robustbench). Finally, we provide theoretical support for our empirical findings.

References in corpus (33)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Deep Residual Learning for Image Recognition
• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• Language Models are Few-Shot Learners
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• AMC: AutoML for Model Compression and Acceleration on Mobile Devices
• EfficientNetV2: Smaller Models and Faster Training
• Understanding deep learning requires rethinking generalization
• Pruning Filters for Efficient ConvNets
• AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty
• Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights
• Learning Structured Sparsity in Deep Neural Networks
• The State of Sparsity in Deep Neural Networks
• Variational Dropout Sparsifies Deep Neural Networks
• Comparing Rewinding and Fine-tuning in Neural Network Pruning
• Learning Sparse Neural Networks through $L_0$ Regularization
• Uncovering the Limits of Adversarial Training against Norm-Bounded Adversarial Examples
• Exploring Sparsity in Recurrent Neural Networks
• RobustBench: a standardized adversarial robustness benchmark
• Sharpness-Aware Minimization for Efficiently Improving Generalization
• What Do Compressed Deep Neural Networks Forget?
• How Can We Be So Dense? The Benefits of Using Highly Sparse Representations
• Lost in Pruning: The Effects of Pruning Neural Networks beyond Test Accuracy
• Optimal Lottery Tickets via SubsetSum: Logarithmic Over-Parameterization is Sufficient
• On the effectiveness of adversarial training against common corruptions
• Can Subnetwork Structure be the Key to Out-of-Distribution Generalization?
• SiPPing Neural Networks: Sensitivity-informed Provable Pruning of Neural Networks
• Achieving Adversarial Robustness via Sparsity
• Multi-Prize Lottery Ticket Hypothesis: Finding Accurate Binary Neural Networks by Pruning A Randomly Weighted Network
• Calibrate and Prune: Improving Reliability of Lottery Tickets Through Prediction Calibration
• Semi-supervised on-device neural network adaptation for remote and portable laser-induced breakdown spectroscopy
• Defending Against Image Corruptions Through Adversarial Augmentations

Cited by in corpus (1)

• Certified Adversarial Defenses Meet Out-of-Distribution Corruptions: Benchmarking Robustness and Simple Baselines