CAPE: Context-Aware Private Embeddings for Private Language Learning

Abstract

Deep learning-based language models have achieved state-of-the-art results in a number of applications including sentiment analysis, topic labelling, intent classification and others. Obtaining text representations or embeddings using these models presents the possibility of encoding personally identifiable information learned from language and context cues that may present a risk to reputation or privacy. To ameliorate these issues, we propose Context-Aware Private Embeddings (CAPE), a novel approach which preserves privacy during training of embeddings. To maintain the privacy of text representations, CAPE applies calibrated noise through differential privacy, preserving the encoded semantic links while obscuring sensitive information. In addition, CAPE employs an adversarial training regime that obscures identified private variables. Experimental results demonstrate that the proposed approach reduces private information leakage better than either single intervention.

arXiv commentAccepted into EMNLP21 main conference

References in corpus (4)

• The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks
• Stolen Memories: Leveraging Model Memorization for Calibrated White-Box Membership Inference
• Domain-Adversarial Training of Neural Networks
• Scalable Differential Privacy with Certified Robustness in Adversarial Learning