Towards a Robust and Trustworthy Machine Learning System Development: An Engineering Perspective

Abstract

While Machine Learning (ML) technologies are widely adopted in many mission critical fields to support intelligent decision-making, concerns remain about system resilience against ML-specific security attacks and privacy breaches as well as the trust that users have in these systems. In this article, we present our recent systematic and comprehensive survey on the state-of-the-art ML robustness and trustworthiness from a security engineering perspective, focusing on the problems in system threat analysis, design and evaluation faced in developing practical machine learning applications, in terms of robustness and user trust. Accordingly, we organize the presentation of this survey intended to facilitate the convey of the body of knowledge from this angle. We then describe a metamodel we created that represents the body of knowledge in a standard and visualized way. We further illustrate how to leverage the metamodel to guide a systematic threat analysis and security design process which extends and scales up the classic process. Finally, we propose the future research directions motivated by our findings. Our work differs itself from the existing surveys by (i) exploring the fundamental principles and best practices to support robust and trustworthy ML system development, and (ii) studying the interplay of robustness and user trust in the context of ML systems. We expect this survey provides a big picture for machine learning security practitioners.

arXiv comment20 pages (58 pages pre-print), 6 figures

References in corpus (35)

• Explaining and Harnessing Adversarial Examples
• Ensemble Adversarial Training: Attacks and Defenses
• Principles alone cannot guarantee ethical AI
• One pixel attack for fooling deep neural networks
• Unsupervised Data Augmentation for Consistency Training
• RAPPOR: Randomized Aggregatable Privacy-Preserving Ordinal Response
• Wild Patterns: Ten Years After the Rise of Adversarial Machine Learning
• How To Backdoor Federated Learning
• Generalizing from a Few Examples: A Survey on Few-Shot Learning
• Stealing Machine Learning Models via Prediction APIs
• Governing autonomous vehicles: emerging responses for safety, liability, privacy, cybersecurity, and industry risks
• Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks
• On Evaluating Adversarial Robustness
• Certified Defenses against Adversarial Examples
• Security Evaluation of Pattern Classifiers under Attack
• Understanding Adversarial Training: Increasing Local Stability of Neural Nets through Robust Optimization
• Scalable Private Learning with PATE
• Adversarial Examples: Opportunities and Challenges
• Keeping the Bad Guys Out: Protecting and Vaccinating Deep Learning with JPEG Compression
• Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims
• Poisoning Attacks to Graph-Based Recommender Systems
• Towards the Science of Security and Privacy in Machine Learning
• Autonomous Vehicles for Smart and Sustainable Cities: An In-Depth Exploration of Privacy and Cybersecurity Implications
• Foveation-based Mechanisms Alleviate Adversarial Examples
• Data Poisoning Attacks to Deep Learning Based Recommender Systems
• AttriGuard: A Practical Defense Against Attribute Inference Attacks via Adversarial Machine Learning
• UPSET and ANGRI : Breaking High Performance Image Classifiers
• DeepCloak: Masking Deep Neural Network Models for Robustness Against Adversarial Samples
• Discretization based Solutions for Secure Machine Learning against Adversarial Attacks
• A Survey of Game Theoretic Approaches for Adversarial Machine Learning in Cybersecurity Tasks
• Ontologies for Network Security and Future Challenges
• Summoning Demons: The Pursuit of Exploitable Bugs in Machine Learning
• Strong Black-box Adversarial Attacks on Unsupervised Machine Learning Models
• New CleverHans Feature: Better Adversarial Robustness Evaluations with Attack Bundling
• Hacking Neural Networks: A Short Introduction