Triple Wins: Boosting Accuracy, Robustness and Efficiency Together by Enabling Input-Adaptive Inference

Abstract

Deep networks were recently suggested to face the odds between accuracy (on clean natural images) and robustness (on adversarially perturbed images) (Tsipras et al., 2019). Such a dilemma is shown to be rooted in the inherently higher sample complexity (Schmidt et al., 2018) and/or model capacity (Nakkiran, 2019), for learning a high-accuracy and robust classifier. In view of that, give a classification task, growing the model capacity appears to help draw a win-win between accuracy and robustness, yet at the expense of model size and latency, therefore posing challenges for resource-constrained applications. Is it possible to co-design model accuracy, robustness and efficiency to achieve their triple wins? This paper studies multi-exit networks associated with input-adaptive efficient inference, showing their strong promise in achieving a "sweet point" in cooptimizing model accuracy, robustness and efficiency. Our proposed solution, dubbed Robust Dynamic Inference Networks (RDI-Nets), allows for each input (either clean or adversarial) to adaptively choose one of the multiple output layers (early branches or the final one) to output its prediction. That multi-loss adaptivity adds new variations and flexibility to adversarial attacks and defenses, on which we present a systematical investigation. We show experimentally that by equipping existing backbones with such robust adaptive inference, the resulting RDI-Nets can achieve better accuracy and robustness, yet with over 30% computational savings, compared to the defended original models.

Published on ICLR 2020

References in corpus (11)

• MobileNetV2: Inverted Residuals and Linear Bottlenecks
• Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networks
• Theoretically Principled Trade-off between Robustness and Accuracy
• Certified Adversarial Robustness via Randomized Smoothing
• Multi-Scale Dense Networks for Resource Efficient Image Classification
• Adversarial Transformation Networks: Learning to Generate Adversarial Examples
• Sparse DNNs with Improved Adversarial Robustness
• Deep $k$-Means: Re-Training and Parameter Sharing with Harder Cluster Assignments for Compressing Deep Convolutions
• Adversarial Robustness May Be at Odds With Simplicity
• Ensemble Methods as a Defense to Adversarial Perturbations Against Deep Neural Networks
• Towards Understanding Adversarial Examples Systematically: Exploring Data Size, Task and Model Factors

Cited by in corpus (14)

• Adaptive Inference through Early-Exit Networks: Design, Challenges and Directions
• IA-RED${}^2$: Interpretability-Aware Redundancy Reduction for Vision Transformers
• BERT Loses Patience: Fast and Robust Inference with Early Exit
• Dynamic Neural Networks: A Survey
• Adversarial Robustness: From Self-Supervised Pre-Training to Fine-Tuning
• Opportunities and Challenges in Deep Learning Adversarial Robustness: A Survey
• Once-for-All Adversarial Training: In-Situ Tradeoff between Robustness and Accuracy for Free
• A Panda? No, It's a Sloth: Slowdown Attacks on Adaptive Multi-Exit Neural Network Inference
• Adversarial Feature Augmentation and Normalization for Visual Recognition
• Data-Efficient GAN Training Beyond (Just) Augmentations: A Lottery Ticket Perspective
• Universal Adder Neural Networks
• Can 3D Adversarial Logos Cloak Humans?
• Learning Transferable 3D Adversarial Cloaks for Deep Trained Detectors
• SmartDeal: Re-Modeling Deep Network Weights for Efficient Inference and Training