On the Privacy Risks of Deploying Recurrent Neural Networks in Machine Learning Models

Abstract

We study the privacy implications of training recurrent neural networks (RNNs) with sensitive training datasets. Considering membership inference attacks (MIAs), which aim to infer whether or not specific data records have been used in training a given machine learning model, we provide empirical evidence that a neural network's architecture impacts its vulnerability to MIAs. In particular, we demonstrate that RNNs are subject to a higher attack accuracy than feed-forward neural network (FFNN) counterparts. Additionally, we study the effectiveness of two prominent mitigation methods for preempting MIAs, namely weight regularization and differential privacy. For the former, we empirically demonstrate that RNNs may only benefit from weight regularization marginally as opposed to FFNNs. For the latter, we find that enforcing differential privacy through either of the following two methods leads to a less favorable privacy-utility trade-off in RNNs than alternative FFNNs: (i) adding Gaussian noise to the gradients calculated during training as a part of the so-called DP-SGD algorithm and (ii) adding Gaussian noise to the trainable parameters as a part of a post-training mechanism that we propose. As a result, RNNs can also be less amenable to mitigation methods, bringing us to the conclusion that the privacy risks pertaining to the recurrent architecture are higher than the feed-forward counterparts.

arXiv commentUnder Double-Blind Review

References in corpus (25)

• Deep Learning with Differential Privacy
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Trust Region Policy Optimization
• Convolutional Sequence to Sequence Learning
• Soft Actor-Critic Algorithms and Applications
• Comprehensive Privacy Analysis of Deep Learning: Passive and Active White-box Inference Attacks against Centralized and Federated Learning
• Language Modeling with Gated Convolutional Networks
• Learning Differentially Private Recurrent Language Models
• Differentially Private Empirical Risk Minimization
• Protection Against Reconstruction and Its Applications in Private Federated Learning
• ReNet: A Recurrent Neural Network Based Alternative to Convolutional Networks
• Spatially-sparse convolutional neural networks
• Understanding Membership Inferences on Well-Generalized Learning Models
• White-box vs Black-box: Bayes Optimal Strategies for Membership Inference
• Gaussian Differential Privacy
• Privacy in Deep Learning: A Survey
• Stable Recurrent Models
• Functional Mechanism: Regression Analysis under Differential Privacy
• Neural Proximal/Trust Region Policy Optimization Attains Globally Optimal Policy
• Recurrent Reinforcement Learning: A Hybrid Approach
• Membership Inference Attacks on Machine Learning: A Survey
• CNN Filter DB: An Empirical Investigation of Trained Convolutional Filters
• A Differentially Private Framework for Deep Learning with Convexified Loss Functions
• Pain-Free Random Differential Privacy with Sensitivity Sampling
• Quantifying Membership Inference Vulnerability via Generalization Gap and Other Model Metrics