Stealing Deep Reinforcement Learning Models for Fun and Profit
arXiv:2006.05032
Abstract
This paper presents the first model extraction attack against Deep Reinforcement Learning (DRL), which enables an external adversary to precisely recover a black-box DRL model only from its interaction with the environment. Model extraction attacks against supervised Deep Learning models have been widely studied. However, those techniques cannot be applied to the reinforcement learning scenario due to DRL models' high complexity, stochasticity and limited observable information. We propose a novel methodology to overcome the above challenges. The key insight of our approach is that the process of DRL model extraction is equivalent to imitation learning, a well-established solution to learn sequential decision-making policies. Based on this observation, our methodology first builds a classifier to reveal the training algorithm family of the targeted black-box DRL model only based on its predicted actions, and then leverages state-of-the-art imitation learning techniques to replicate the model from the identified algorithm family. Experimental results indicate that our methodology can effectively recover the DRL models with high fidelity and accuracy. We also demonstrate two use cases to show that our model extraction attack can (1) significantly improve the success rate of adversarial attacks, and (2) steal DRL models stealthily even they are protected by DNN watermarks. These pose a severe threat to the intellectual property and privacy protection of DRL applications.
References in corpus (17)
- Continuous control with deep reinforcement learning
- Embedding Watermarks into Deep Neural Networks
- Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization
- Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation
- Deep Q-learning from Demonstrations
- MagNet: a Two-Pronged Defense against Adversarial Examples
- Sample Efficient Actor-Critic with Experience Replay
- Copycat CNN: Stealing Knowledge by Persuading Confession with Random Non-Labeled Data
- Neural Network Model Extraction Attacks in Edge Devices by Hearing Architectural Hints
- Stealing Neural Networks via Timing Side Channels
- Security Analysis of Deep Neural Networks Operating in the Presence of Cache Side-Channel Attacks
- Characterizing Attacks on Deep Reinforcement Learning
- Cache Telepathy: Leveraging Shared Resource Attacks to Learn DNN Architectures
- A framework for the extraction of Deep Neural Networks by leveraging public data
- Exploring Connections Between Active Learning and Model Extraction
- Optimal Attacks on Reinforcement Learning Policies
- Sequential Triggers for Watermarking of Deep Reinforcement Learning Policies