Study of positional encoding approaches for Audio Spectrogram Transformers
arXiv:2110.06999 · doi:10.1109/ICASSP43922.2022.9747742
Abstract
Transformers have revolutionized the world of deep learning, specially in the field of natural language processing. Recently, the Audio Spectrogram Transformer (AST) was proposed for audio classification, leading to state of the art results in several datasets. However, in order for ASTs to outperform CNNs, pretraining with ImageNet is needed. In this paper, we study one component of the AST, the positional encoding, and propose several variants to improve the performance of ASTs trained from scratch, without ImageNet pretraining. Our best model, which incorporates conditional positional encodings, significantly improves performance on Audioset and ESC-50 compared to the original AST.
Submitted to ICASSP 2022. 5 pages, 3 figures
References in corpus (11)
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Zero-Shot Text-to-Image Generation
- ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases
- A Model of Inductive Bias Learning
- Conditional Positional Encodings for Vision Transformers
- RoFormer: Enhanced Transformer with Rotary Position Embedding
- PSLA: Improving Audio Tagging with Pretraining, Sampling, Labeling, and Aggregation
- Audio Transformers
- Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation
- A Sequential Self Teaching Approach for Improving Generalization in Sound Event Recognition
- A Framework for Generative and Contrastive Learning of Audio Representations