An Analysis of Linear Complexity Attention Substitutes with BEST-RQ

Ryan Whetten, Titouan Parcollet, Adel Moumen +2

Self-Supervised Learning (SSL) has proven to be effective in various domains, including speech processing. However, SSL is computationally and memory expensive. This is in part due…