Cost-effective Deployment of BERT Models in Serverless Environment

Abstract

In this study we demonstrate the viability of deploying BERT-style models to serverless environments in a production setting. Since the freely available pre-trained models are too large to be deployed in this way, we utilize knowledge distillation and fine-tune the models on proprietary datasets for two real-world tasks: sentiment analysis and semantic textual similarity. As a result, we obtain models that are tuned for a specific domain and deployable in serverless environments. The subsequent performance analysis shows that this solution results in latency levels acceptable for production use and that it is also a cost-effective approach for small-to-medium size deployments of BERT models, all without any infrastructure overhead.

NAACL-HLT 2021 Industry Track Camera Ready

References in corpus (5)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Distilling the Knowledge in a Neural Network
• MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices
• No Training Required: Exploring Random Encoders for Sentence Classification
• Exploiting BERT for End-to-End Aspect-based Sentiment Analysis