AST: Audio Spectrogram Transformer

Abstract

In the past decade, convolutional neural networks (CNNs) have been widely adopted as the main building block for end-to-end audio classification models, which aim to learn a direct mapping from audio spectrograms to corresponding labels. To better capture long-range global context, a recent trend is to add a self-attention mechanism on top of the CNN, forming a CNN-attention hybrid model. However, it is unclear whether the reliance on a CNN is necessary, and if neural networks purely based on attention are sufficient to obtain good performance in audio classification. In this paper, we answer the question by introducing the Audio Spectrogram Transformer (AST), the first convolution-free, purely attention-based model for audio classification. We evaluate AST on various audio classification benchmarks, where it achieves new state-of-the-art results of 0.485 mAP on AudioSet, 95.6% accuracy on ESC-50, and 98.1% accuracy on Speech Commands V2.

arXiv commentAccepted at Interspeech 2021. Code at https://github.com/YuanGongND/ast

References in corpus (7)

• SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
• Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition
• Learning from Between-class Examples for Deep Sound Recognition
• PSLA: Improving Audio Tagging with Pretraining, Sampling, Labeling, and Aggregation
• Streaming keyword spotting on mobile devices
• Rethinking CNN Models for Audio Classification
• MatchboxNet: 1D Time-Channel Separable Convolutional Neural Network Architecture for Speech Commands Recognition

Cited by in corpus (8)

• Attention Bottlenecks for Multimodal Fusion
• Pruning vs XNOR-Net: A Comprehensive Study of Deep Learning for Audio Classification on Edge-devices
• MT3: Multi-Task Multitrack Music Transcription
• Towards Learning Universal Audio Representations
• Attention-Free Keyword Spotting
• Audio Spectral Enhancement: Leveraging Autoencoders for Low Latency Reconstruction of Long, Lossy Audio Sequences
• Pathologies in priors and inference for Bayesian transformers
• Audiomer: A Convolutional Transformer For Keyword Spotting