Audiomer: A Convolutional Transformer For Keyword Spotting

Abstract

Transformers have seen an unprecedented rise in Natural Language Processing and Computer Vision tasks. However, in audio tasks, they are either infeasible to train due to extremely large sequence length of audio waveforms or incur a performance penalty when trained on Fourier-based features. In this work, we introduce an architecture, Audiomer, where we combine 1D Residual Networks with Performer Attention to achieve state-of-the-art performance in keyword spotting with raw audio waveforms, outperforming all previous methods while being computationally cheaper and parameter-efficient. Additionally, our model has practical advantages for speech processing, such as inference on arbitrarily long audio clips owing to the absence of positional encoding. The code is available at https://github.com/The-Learning-Machines/Audiomer-PyTorch.

The results and claims made are incorrect due to data leakage and an erroneous split of datasets

References in corpus (7)

• Linformer: Self-Attention with Linear Complexity
• Conformer: Convolution-augmented Transformer for Speech Recognition
• Escaping the Big Data Paradigm with Compact Transformers
• CvT: Introducing Convolutions to Vision Transformers
• Keyword Transformer: A Self-Attention Model for Keyword Spotting
• Fastformer: Additive Attention Can Be All You Need
• AST: Audio Spectrogram Transformer