Masked Autoencoders with Multi-Window Local-Global Attention Are Better Audio Learners

Sarthak Yadav, Sergios Theodoridis, Lars Kai Hansen +1

In this work, we propose a Multi-Window Masked Autoencoder (MW-MAE) fitted with a novel Multi-Window Multi-Head Attention (MW-MHA) module that facilitates the modelling of local-gl…