The TrojAI Software Framework: An OpenSource tool for Embedding Trojans into Deep Learning Models

Abstract

In this paper, we introduce the TrojAI software framework, an open source set of Python tools capable of generating triggered (poisoned) datasets and associated deep learning (DL) models with trojans at scale. We utilize the developed framework to generate a large set of trojaned MNIST classifiers, as well as demonstrate the capability to produce a trojaned reinforcement-learning model using vector observations. Results on MNIST show that the nature of the trigger, training batch size, and dataset poisoning percentage all affect successful embedding of trojans. We test Neural Cleanse against the trojaned MNIST models and successfully detect anomalies in the trained models approximately $18\%$ of the time. Our experiments and workflow indicate that the TrojAI software framework will enable researchers to easily understand the effects of various configurations of the dataset and training hyperparameters on the generated trojaned deep learning model, and can be used to rapidly and comprehensively test new trojan detection methods.

arXiv comment8 pages, 16 figures

References in corpus (7)

• Adam: A Method for Stochastic Optimization
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain
• PoTrojan: powerful neural-level trojan designs in deep learning models
• Unsupervised State Representation Learning in Atari
• TrojDRL: Trojan Attacks on Deep Reinforcement Learning Agents
• Design of intentional backdoors in sequential models

Cited by in corpus (6)

• Backdoor Attacks and Countermeasures on Deep Learning: A Comprehensive Review
• Unsolved Problems in ML Safety
• TOP: Backdoor Detection in Neural Networks via Transferability of Perturbation
• Poisoning Deep Reinforcement Learning Agents with In-Distribution Triggers
• TAD: Trigger Approximation based Black-box Trojan Detection for AI
• Gradient Shaping: Enhancing Backdoor Attack Against Reverse Engineering