Masked Autoencoders with Multi-Window Local-Global Attention Are Better Audio Learners

Abstract

In this work, we propose a Multi-Window Masked Autoencoder (MW-MAE) fitted with a novel Multi-Window Multi-Head Attention (MW-MHA) module that facilitates the modelling of local-global interactions in every decoder transformer block through attention heads of several distinct local and global windows. Empirical results on ten downstream audio tasks show that MW-MAEs consistently outperform standard MAEs in overall performance and learn better general-purpose audio representations, along with demonstrating considerably better scaling characteristics. Investigating attention distances and entropies reveals that MW-MAE encoders learn heads with broader local and global attention. Analyzing attention head feature representations through Projection Weighted Canonical Correlation Analysis (PWCCA) shows that attention heads with the same window sizes across the decoder layers of the MW-MAE learn correlated feature representations which enables each block to independently capture local and global information, leading to a decoupled decoder feature hierarchy. Code for feature extraction and downstream experiments along with pre-trained models will be released publically.

References in corpus (22)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• A Simple Framework for Contrastive Learning of Visual Representations
• Representation Learning with Contrastive Predictive Coding
• Bootstrap your own latent: A new approach to self-supervised Learning
• WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
• Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition
• BEiT: BERT Pre-Training of Image Transformers
• Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders
• Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders
• vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations
• Masked Autoencoders As Spatiotemporal Learners
• data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language
• Efficient Training of Audio Transformers with Patchout
• Masked Autoencoders that Listen
• BEATs: Audio Pre-Training with Acoustic Tokenizers
• Classification vs. Regression in Supervised Learning for Single Channel Speaker Count Estimation
• HEAR: Holistic Evaluation of Audio Representations
• Time-Contrastive Learning Based Deep Bottleneck Features for Text-Dependent Speaker Verification
• Masked Spectrogram Modeling using Masked Autoencoders for Learning General-purpose Audio Representation
• TorchAudio: Building Blocks for Audio and Speech Processing
• Masked World Models for Visual Control
• BYOL-S: Learning Self-supervised Speech Representations by Bootstrapping