Sisyphus: A Cautionary Tale of Using Low-Degree Polynomial Activations in Privacy-Preserving Deep Learning
arXiv:2107.12342
Abstract
Privacy concerns in client-server machine learning have given rise to private inference (PI), where neural inference occurs directly on encrypted inputs. PI protects clients' personal data and the server's intellectual property. A common practice in PI is to use garbled circuits to compute nonlinear functions privately, namely ReLUs. However, garbled circuits suffer from high storage, bandwidth, and latency costs. To mitigate these issues, PI-friendly polynomial activation functions have been employed to replace ReLU. In this work, we ask: Is it feasible to substitute all ReLUs with low-degree polynomial activation functions for building deep, privacy-friendly neural networks? We explore this question by analyzing the challenges of substituting ReLUs with polynomials, starting with simple drop-and-replace solutions to novel, more involved replace-and-retrain strategies. We examine the limitations of each method and provide commentary on the use of polynomial activation functions for PI. We find all evaluated solutions suffer from the escaping activation problem: forward activation values inevitably begin to expand at an exponential rate away from stable regions of the polynomials, which leads to exploding values (NaNs) or poor approximations.
arXiv commentAccepted to PPML (ACM CCS) 2021
References in corpus (19)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- Deep Learning with Differential Privacy
- Gaussian Error Linear Units (GELUs)
- Federated Learning: Strategies for Improving Communication Efficiency
- Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
- Empirical Evaluation of Rectified Activations in Convolutional Network
- Towards Federated Learning at Scale: System Design
- Searching for Activation Functions
- GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration
- CrypTFlow2: Practical 2-Party Secure Inference
- CryptoDL: Deep Neural Networks over Encrypted Data
- Slalom: Fast, Verifiable and Private Execution of Neural Networks in Trusted Hardware
- BoTorch: A Framework for Efficient Monte-Carlo Bayesian Optimization
- Chiron: Privacy-preserving Machine Learning as a Service
- Faster CryptoNets: Leveraging Sparsity for Real-World Encrypted Inference
- Padé Activation Units: End-to-end Learning of Flexible Activation Functions in Deep Networks
- Precise Approximation of Convolutional Neural Networks for Homomorphically Encrypted Data