Privacy Loss in Apple's Implementation of Differential Privacy on MacOS 10.12

Abstract

In June 2016, Apple announced that it will deploy differential privacy for some user data collection in order to ensure privacy of user data, even from Apple. The details of Apple's approach remained sparse. Although several patents have since appeared hinting at the algorithms that may be used to achieve differential privacy, they did not include a precise explanation of the approach taken to privacy parameter choice. Such choice and the overall approach to privacy budget use and management are key questions for understanding the privacy protections provided by any deployment of differential privacy. In this work, through a combination of experiments, static and dynamic code analysis of macOS Sierra (Version 10.12) implementation, we shed light on the choices Apple made for privacy budget management. We discover and describe Apple's set-up for differentially private data processing, including the overall data pipeline, the parameters used for differentially private perturbation of each piece of data, and the frequency with which such data is sent to Apple's servers. We find that although Apple's deployment ensures that the (differential) privacy loss per each datum submitted to its servers is $1$ or $2$, the overall privacy loss permitted by the system is significantly higher, as high as $16$ per day for the four initially announced applications of Emojis, New words, Deeplinks and Lookup Hints. Furthermore, Apple renews the privacy budget available every day, which leads to a possible privacy loss of 16 times the number of days since user opt-in to differentially private data collection for those four applications. We advocate that in order to claim the full benefits of differentially private data collection, Apple must give full transparency of its implementation, enable user choice in areas related to privacy loss, and set meaningful defaults on the privacy loss permitted.

References in corpus (4)

• Differential Privacy: An Economic Method for Choosing Epsilon
• Proving Differential Privacy via Probabilistic Couplings
• Proving differential privacy in Hoare logic
• Privacy in Information-Rich Intelligent Infrastructure

Cited by in corpus (48)

• Mobile Sensor Data Anonymization
• The Long Road to Computational Location Privacy: A Survey
• Context-Aware Generative Adversarial Privacy
• A Comprehensive Survey on Local Differential Privacy Toward Data Statistics and Analysis
• Efficient privacy preservation of big data for accurate data mining
• LDP-IDS: Local Differential Privacy for Infinite Data Streams
• Evaluating Differentially Private Machine Learning in Practice
• Efficient Data Perturbation for Privacy Preserving and Accurate Data Stream Mining
• Privacy-Preserving Blockchain-Based Federated Learning for IoT Devices
• Comparative Study of Differentially Private Data Synthesis Methods
• Collecting and Analyzing Multidimensional Data with Local Differential Privacy
• Encode, Shuffle, Analyze Privacy Revisited: Formalizations and Empirical Evaluation
• Local Differential Privacy: a tutorial
• Pointwise Maximal Leakage
• Differentially-Private "Draw and Discard" Machine Learning
• Local Differential Privacy for Evolving Data
• BLENDER: Enabling Local Search with a Hybrid Differential Privacy Model
• Reviewing and Improving the Gaussian Mechanism for Differential Privacy
• Differential Privacy for Government Agencies -- Are We There Yet?
• Information-theoretic metrics for Local Differential Privacy protocols
• Estimating Smooth GLM in Non-interactive Local Differential Privacy Model with Public Unlabeled Data
• Robust and Differentially Private Mean Estimation
• Privacy-Preserving Blockchain Based Federated Learning with Differential Data Sharing
• Chasing Your Long Tails: Differentially Private Prediction in Health Care Settings
• Aggregating Votes with Local Differential Privacy: Usefulness, Soundness vs. Indistinguishability
• An Incentive Mechanism for Trading Personal Data in Data Markets
• A Blockchain-Based Approach for Saving and Tracking Differential-Privacy Cost
• An Analysis of the Deployment of Models Trained on Private Tabular Synthetic Data: Unexpected Surprises
• LIA: Privacy-Preserving Data Quality Evaluation in Federated Learning Using a Lazy Influence Approximation
• Context-aware Data Aggregation with Localized Information Privacy
• Differentially Private High Dimensional Sparse Covariance Matrix Estimation
• Effective and Privacy preserving Tabular Data Synthesizing
• Privately Publishable Per-instance Privacy
• Local Differential Privacy in Decentralized Optimization
• Locally private online change point detection
• Local Information Privacy and Its Application to Privacy-Preserving Data Aggregation
• Noise Variance Optimization in Differential Privacy: A Game-Theoretic Approach Through Per-Instance Differential Privacy
• Free Gap Information from the Differentially Private Sparse Vector and Noisy Max Mechanisms
• Privacy-Aware Rejection Sampling
• Nebula: Efficient, Private and Accurate Histogram Estimation
• "I inherently just trust that it works": Investigating Mental Models of Open-Source Libraries for Differential Privacy
• Differential Privacy for Evolving Almost-Periodic Datasets with Continual Linear Queries: Application to Energy Data Privacy
• Empirical Risk Minimization in Non-interactive Local Differential Privacy: Efficiency and High Dimensional Case
• Constructing Privacy Channels from Information Channels
• Privacy-Preserving Public Release of Datasets for Support Vector Machine Classification
• Taking a Lesson from Quantum Particles for Statistical Data Privacy
• Revenue Attribution on iOS 14 using Conversion Values in F2P Games
• Achieving Dalenius' Goal of Data Privacy with Practical Assumptions