Compressed Deep Networks: Goodbye SVD, Hello Robust Low-Rank Approximation

Abstract

A common technique for compressing a neural network is to compute the $k$-rank ${\ell }_2$ approximation $A_{k,2}$ of the matrix $A\in {\mathbb{R}}^{n\times d}$ that corresponds to a fully connected layer (or embedding layer). Here, $d$ is the number of the neurons in the layer, $n$ is the number in the next one, and $A_{k,2}$ can be stored in $O((n+d)k)$ memory instead of $O(nd)$. This ${\ell }_2$-approximation minimizes the sum over every entry to the power of $p=2$ in the matrix $A-A_{k,2}$, among every matrix $A_{k,2}\in {\mathbb{R}}^{n\times d}$ whose rank is $k$. While it can be computed efficiently via SVD, the ${\ell }_2$-approximation is known to be very sensitive to outliers ("far-away" rows). Hence, machine learning uses e.g. Lasso Regression, ${\ell }_1$-regularization, and ${\ell }_1$-SVM that use the ${\ell }_1$-norm. This paper suggests to replace the $k$-rank ${\ell }_2$ approximation by ${\ell }_p$, for $p\in [1,2]$. We then provide practical and provable approximation algorithms to compute it for any $p\ge 1$, based on modern techniques in computational geometry. Extensive experimental results on the GLUE benchmark for compressing BERT, DistilBERT, XLNet, and RoBERTa confirm this theoretical advantage. For example, our approach achieves $28\%$ compression of RoBERTa's embedding layer with only $0.63\%$ additive drop in the accuracy (without fine-tuning) in average over all tasks in GLUE, compared to $11\%$ drop using the existing ${\ell }_2$-approximation. Open code is provided for reproducing and extending our results.

References in corpus (19)

• Distributed Representations of Words and Phrases and their Compositionality
• The NumPy array: a structure for efficient numerical computation
• Distributed Representations of Sentences and Documents
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Training Deep Nets with Sublinear Memory Cost
• Semi-supervised Sequence Learning
• Q8BERT: Quantized 8Bit BERT
• Distilling Task-Specific Knowledge from BERT into Simple Neural Networks
• Reducing Transformer Depth on Demand with Structured Dropout
• MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices
• Structured Pruning of a BERT-based Question Answering Model
• Optimal approximate matrix product in terms of stable rank
• Reweighted Proximal Pruning for Large-Scale Language Representation
• Dimensionality Reduction of Massive Sparse Datasets Using Coresets
• MKD: a Multi-Task Knowledge Distillation Approach for Pretrained Language Models
• Introduction to Coresets: Accurate Coresets
• Distilling BERT into Simple Neural Networks with Unlabeled Transfer Data
• Coresets for Near-Convex Functions
• Faster PAC Learning and Smaller Coresets via Smoothed Analysis

Cited by in corpus (3)

• Compressing Neural Networks: Towards Determining the Optimal Layer-wise Decomposition
• BERMo: What can BERT learn from ELMo?
• Deep Learning Meets Projective Clustering