Masked Autoencoders that Listen

Abstract

This paper studies a simple extension of image-based Masked Autoencoders (MAE) to self-supervised representation learning from audio spectrograms. Following the Transformer encoder-decoder design in MAE, our Audio-MAE first encodes audio spectrogram patches with a high masking ratio, feeding only the non-masked tokens through encoder layers. The decoder then re-orders and decodes the encoded context padded with mask tokens, in order to reconstruct the input spectrogram. We find it beneficial to incorporate local window attention in the decoder, as audio spectrograms are highly correlated in local time and frequency bands. We then fine-tune the encoder with a lower masking ratio on target datasets. Empirically, Audio-MAE sets new state-of-the-art performance on six audio and speech classification tasks, outperforming other recent models that use external supervised pre-training. The code and models will be at https://github.com/facebookresearch/AudioMAE.

Accepted at NeurIPS 2022

Cited by in corpus (14)

• DINOv2: Learning Robust Visual Features without Supervision
• HiCMAE: Hierarchical Contrastive Masked Autoencoder for Self-Supervised Audio-Visual Emotion Recognition
• Whisper-AT: Noise-Robust Automatic Speech Recognizers are Also Strong General Audio Event Taggers
• Masked Autoencoders for Microscopy are Scalable Learners of Cellular Biology
• TMac: Temporal Multi-Modal Graph Learning for Acoustic Event Classification
• Self-supervised language learning from raw audio: Lessons from the Zero Resource Speech Challenge
• ASiT: Local-Global Audio Spectrogram vIsion Transformer for Event Classification
• MultiMAE-DER: Multimodal Masked Autoencoder for Dynamic Emotion Recognition
• Advancing Multi-grained Alignment for Contrastive Language-Audio Pre-training
• A vector quantized masked autoencoder for audiovisual speech emotion recognition
• MDT-A2G: Exploring Masked Diffusion Transformers for Co-Speech Gesture Generation
• Multimodal Laryngoscopic Video Analysis for Assisted Diagnosis of Vocal Fold Paralysis
• Bridging the Gap Between Semantic and User Preference Spaces for Multi-modal Music Representation Learning
• Masked Autoencoders with Multi-Window Local-Global Attention Are Better Audio Learners