Adversarially Robust and Explainable Model Compression with On-Device Personalization for Text Classification

Abstract

On-device Deep Neural Networks (DNNs) have recently gained more attention due to the increasing computing power of the mobile devices and the number of applications in Computer Vision (CV), Natural Language Processing (NLP), and Internet of Things (IoTs). Unfortunately, the existing efficient convolutional neural network (CNN) architectures designed for CV tasks are not directly applicable to NLP tasks and the tiny Recurrent Neural Network (RNN) architectures have been designed primarily for IoT applications. In NLP applications, although model compression has seen initial success in on-device text classification, there are at least three major challenges yet to be addressed: adversarial robustness, explainability, and personalization. Here we attempt to tackle these challenges by designing a new training scheme for model compression and adversarial robustness, including the optimization of an explainable feature mapping objective, a knowledge distillation objective, and an adversarially robustness objective. The resulting compressed model is personalized using on-device private training data via fine-tuning. We perform extensive experiments to compare our approach with both compact RNN (e.g., FastGRNN) and compressed RNN (e.g., PRADO) architectures in both natural and adversarial NLP test settings.

arXiv comment8 pages, 4 figures

References in corpus (22)

• Distilling the Knowledge in a Neural Network
• Intriguing properties of neural networks
• Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks
• Deep Text Classification Can be Fooled
• Adversarial Risk and the Dangers of Evaluating Against Weak Attacks
• Deep Learning for Sensor-based Human Activity Recognition: Overview, Challenges and Opportunities
• LEAF: A Benchmark for Federated Settings
• Adversarial Examples for Evaluating Reading Comprehension Systems
• Towards Crafting Text Adversarial Samples
• Factorization tricks for LSTM networks
• Rethinking Softmax Cross-Entropy Loss for Adversarial Robustness
• MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices
• Improving the Generalization of Adversarial Training with Domain Adaptation
• HotFlip: White-Box Adversarial Examples for Text Classification
• Model Compression with Adversarial Robustness: A Unified Optimization Framework
• ProjectionNet: Learning Efficient On-Device Deep Networks Using Neural Projections
• Skip RNN: Learning to Skip State Updates in Recurrent Neural Networks
• Towards Explainable NLP: A Generative Explanation Framework for Text Classification
• Personalized Speech recognition on mobile devices
• Learning to Discriminate Perturbations for Blocking Adversarial Attacks in Text Classification
• Using Distance Estimation and Deep Learning to Simplify Calibration in Food Calorie Measurement
• How Large a Vocabulary Does Text Classification Need? A Variational Approach to Vocabulary Selection