RNAS-CL: Robust Neural Architecture Search by Cross-Layer Knowledge Distillation
arXiv:2301.08092 · doi:10.1007/s11263-024-02133-4
Abstract
Deep Neural Networks are vulnerable to adversarial attacks. Neural Architecture Search (NAS), one of the driving tools of deep neural networks, demonstrates superior performance in prediction accuracy in various machine learning applications. However, it is unclear how it performs against adversarial attacks. Given the presence of a robust teacher, it would be interesting to investigate if NAS would produce robust neural architecture by inheriting robustness from the teacher. In this paper, we propose Robust Neural Architecture Search by Cross-Layer Knowledge Distillation (RNAS-CL), a novel NAS algorithm that improves the robustness of NAS by learning from a robust teacher through cross-layer knowledge distillation. Unlike previous knowledge distillation methods that encourage close student/teacher output only in the last layer, RNAS-CL automatically searches for the best teacher layer to supervise each student layer. Experimental result evidences the effectiveness of RNAS-CL and shows that RNAS-CL produces small and robust neural architecture.
arXiv comment17 pages, 12 figures
References in corpus (24)
- Explaining and Harnessing Adversarial Examples
- Neural Architecture Search with Reinforcement Learning
- Ensemble Adversarial Training: Attacks and Defenses
- Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer
- Towards Deep Learning Models Resistant to Adversarial Attacks
- Theoretically Principled Trade-off between Robustness and Accuracy
- Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks
- MixMatch: A Holistic Approach to Semi-Supervised Learning
- Fast is better than free: Revisiting adversarial training
- Countering Adversarial Images using Input Transformations
- PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search
- ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware
- Formal Guarantees on the Robustness of a Classifier against Adversarial Manipulation
- Parseval Networks: Improving Robustness to Adversarial Examples
- Adversarially Robust Distillation
- Faster CNNs with Direct Sparse Convolutions and Guided Pruning
- Rethinking Softmax Cross-Entropy Loss for Adversarial Robustness
- Model Compression with Adversarial Robustness: A Unified Optimization Framework
- Cream of the Crop: Distilling Prioritized Paths For One-Shot Neural Architecture Search
- ISTA-NAS: Efficient and Consistent Neural Architecture Search by Sparse Coding
- HYDRA: Pruning Adversarially Robust Neural Networks
- Exploring Architectural Ingredients of Adversarially Robust Deep Neural Networks
- When Adversarial Training Meets Vision Transformers: Recipes from Training to Architecture
- Neural Architecture Dilation for Adversarial Robustness