PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination

Abstract

We develop a novel method, called PoWER-BERT, for improving the inference time of the popular BERT model, while maintaining the accuracy. It works by: a) exploiting redundancy pertaining to word-vectors (intermediate encoder outputs) and eliminating the redundant vectors. b) determining which word-vectors to eliminate by developing a strategy for measuring their significance, based on the self-attention mechanism. c) learning how many word-vectors to eliminate by augmenting the BERT model and the loss function. Experiments on the standard GLUE benchmark shows that PoWER-BERT achieves up to 4.5x reduction in inference time over BERT with <1% loss in accuracy. We show that PoWER-BERT offers significantly better trade-off between accuracy and inference time compared to prior methods. We demonstrate that our method attains up to 6.8x reduction in inference time with <1% loss in accuracy when applied over ALBERT, a highly compressed version of BERT. The code for PoWER-BERT is publicly available at https://github.com/IBM/PoWER-BERT.

Accepted at ICML 2020

Cited by in corpus (11)

• Centroid Transformers: Learning to Abstract with Attention
• Pruning Self-attentions into Convolutional Layers in Single Path
• Not All Attention Is All You Need
• Patch Slimming for Efficient Vision Transformers
• Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search
• Magic Pyramid: Accelerating Inference with Early Exiting and Token Pruning
• EL-Attention: Memory Efficient Lossless Attention for Generation
• Token Pooling in Vision Transformers
• Exploiting Sentence-Level Representations for Passage Ranking
• Classification-based Quality Estimation: Small and Efficient Models for Real-world Applications
• Learning Dynamic BERT via Trainable Gate Variables and a Bi-modal Regularizer