A Winning Hand: Compressing Deep Networks Can Improve Out-Of-Distribution Robustness

Abstract

Successful adoption of deep learning (DL) in the wild requires models to be: (1) compact, (2) accurate, and (3) robust to distributional shifts. Unfortunately, efforts towards simultaneously meeting these requirements have mostly been unsuccessful. This raises an important question: Is the inability to create Compact, Accurate, and Robust Deep neural networks (CARDs) fundamental? To answer this question, we perform a large-scale analysis of popular model compression techniques which uncovers several intriguing patterns. Notably, in contrast to traditional pruning approaches (e.g., fine tuning and gradual magnitude pruning), we find that "lottery ticket-style" approaches can surprisingly be used to produce CARDs, including binary-weight CARDs. Specifically, we are able to create extremely compact CARDs that, compared to their larger counterparts, have similar test accuracy and matching (or better) robustness -- simply by pruning and (optionally) quantizing. Leveraging the compactness of CARDs, we develop a simple domain-adaptive test-time ensembling approach (CARD-Decks) that uses a gating module to dynamically select appropriate CARDs from the CARD-Deck based on their spectral-similarity with test samples. The proposed approach builds a "winning hand'' of CARDs that establishes a new state-of-the-art (on RobustBench) on CIFAR-10-C accuracies (i.e., 96.8% standard and 92.75% robust) and CIFAR-100-C accuracies (80.6% standard and 71.3% robust) with better memory usage than non-compressed baselines (pretrained CARDs and CARD-Decks available at https://github.com/RobustBench/robustbench). Finally, we provide theoretical support for our empirical findings.

References in corpus (13)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Language Models are Few-Shot Learners
• Understanding deep learning requires rethinking generalization
• Pruning Filters for Efficient ConvNets
• AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty
• Learning Structured Sparsity in Deep Neural Networks
• The State of Sparsity in Deep Neural Networks
• Comparing Rewinding and Fine-tuning in Neural Network Pruning
• How Can We Be So Dense? The Benefits of Using Highly Sparse Representations
• Lost in Pruning: The Effects of Pruning Neural Networks beyond Test Accuracy
• Can Subnetwork Structure be the Key to Out-of-Distribution Generalization?
• Multi-Prize Lottery Ticket Hypothesis: Finding Accurate Binary Neural Networks by Pruning A Randomly Weighted Network
• Semi-supervised on-device neural network adaptation for remote and portable laser-induced breakdown spectroscopy