ResidualTransformer: Residual Low-Rank Learning with Weight-Sharing for Transformer Layers

Yiming Wang, Jinyu Li

Memory constraint of always-on devices is one of the major concerns when deploying speech processing models on these devices. While larger models trained with sufficiently large am…