nGraph-HE2: A High-Throughput Framework for Neural Network Inference on Encrypted Data

Abstract

In previous work, Boemer et al. introduced nGraph-HE, an extension to the Intel nGraph deep learning (DL) compiler, that enables data scientists to deploy models with popular frameworks such as TensorFlow and PyTorch with minimal code changes. However, the class of supported models was limited to relatively shallow networks with polynomial activations. Here, we introduce nGraph-HE2, which extends nGraph-HE to enable privacy-preserving inference on standard, pre-trained models using their native activation functions and number fields (typically real numbers). The proposed framework leverages the CKKS scheme, whose support for real numbers is friendly to data science, and a client-aided model using a two-party approach to compute activation functions. We first present CKKS-specific optimizations, enabling a 3x-88x runtime speedup for scalar encoding, and doubling the throughput through a novel use of CKKS plaintext packing into complex numbers. Second, we optimize ciphertext-plaintext addition and multiplication, yielding 2.6x-4.2x runtime speedup. Third, we exploit two graph-level optimizations: lazy rescaling and depth-aware encoding, which allow us to significantly improve performance. Together, these optimizations enable state-of-the-art throughput of 1,998 images/s on the CryptoNets network. Using the client-aided model, we also present homomorphic evaluation of (to our knowledge) the largest network to date, namely, pre-trained MobileNetV2 models on the ImageNet dataset, with 60.4\percent/82.7\percent\ top-1/top-5 accuracy and an amortized runtime of 381 ms/image.

References in corpus (6)

• XONN: XNOR-based Oblivious Deep Neural Network Inference
• Faster CryptoNets: Leveraging Sparsity for Real-World Encrypted Inference
• Intel nGraph: An Intermediate Representation, Compiler, and Executor for Deep Learning
• Low Latency Privacy Preserving Inference
• SEALion: a Framework for Neural Network Inference on Encrypted Data
• Toward Scalable Fully Homomorphic Encryption Through Light Trusted Computing Assistance

Cited by in corpus (7)

• HEAAN Demystified: Accelerating Fully Homomorphic Encryption Through Architecture-centric Analysis and Optimization
• Secure Human Action Recognition by Encrypted Neural Network Inference
• Enabling Homomorphically Encrypted Inference for Large DNN Models
• CryptoSPN: Privacy-preserving Sum-Product Network Inference
• Partially Oblivious Neural Network Inference
• ArchNet: Data Hiding Model in Distributed Machine Learning System
• A brief history on Homomorphic learning: A privacy-focused approach to machine learning