Model Compression with Adversarial Robustness: A Unified Optimization Framework

Abstract

Deep model compression has been extensively studied, and state-of-the-art methods can now achieve high compression ratios with minimal accuracy loss. This paper studies model compression through a different lens: could we compress models without hurting their robustness to adversarial attacks, in addition to maintaining accuracy? Previous literature suggested that the goals of robustness and compactness might sometimes contradict. We propose a novel Adversarially Trained Model Compression (ATMC) framework. ATMC constructs a unified constrained optimization formulation, where existing compression means (pruning, factorization, quantization) are all integrated into the constraints. An efficient algorithm is then developed. An extensive group of experiments are presented, demonstrating that ATMC obtains remarkably more favorable trade-off among model size, accuracy and robustness, over currently available alternatives in various settings. The codes are publicly available at: https://github.com/shupenggui/ATMC.

14 pages, NeurIPS 2019. The first two authors Gui and Wang contributed equally and are listed alphabetically

Cited by in corpus (23)

• GhostNets on Heterogeneous Devices via Cheap Operations
• A Survey on Long-Tailed Visual Recognition
• GhostNet: More Features from Cheap Operations
• Inspect, Understand, Overcome: A Survey of Practical Methods for AI Safety
• On the Loss Landscape of Adversarial Training: Identifying Challenges and How to Overcome Them
• Adversarial Robustness: From Self-Supervised Pre-Training to Fine-Tuning
• HYDRA: Pruning Adversarially Robust Neural Networks
• Lost in Pruning: The Effects of Pruning Neural Networks beyond Test Accuracy
• Once-for-All Adversarial Training: In-Situ Tradeoff between Robustness and Accuracy for Free
• Achieving Adversarial Robustness via Sparsity
• DASS: Differentiable Architecture Search for Sparse neural networks
• Robust Predictable Control
• GAN Slimming: All-in-One GAN Compression by A Unified Optimization Framework
• SmartExchange: Trading Higher-cost Memory Storage/Access for Lower-cost Computation
• RNAS-CL: Robust Neural Architecture Search by Cross-Layer Knowledge Distillation
• Lipschitz Bounds and Provably Robust Training by Laplacian Smoothing
• Automatic Neural Network Compression by Sparsity-Quantization Joint Learning: A Constrained Optimization-based Approach
• A Tunable Robust Pruning Framework Through Dynamic Network Rewiring of DNNs
• Can 3D Adversarial Logos Cloak Humans?
• Learning Transferable 3D Adversarial Cloaks for Deep Trained Detectors
• Low-Rank+Sparse Tensor Compression for Neural Networks
• Robustifying ${\ell }_{\infty }$ Adversarial Training to the Union of Perturbation Models
• Adversarially Robust and Explainable Model Compression with On-Device Personalization for Text Classification