Tiny Transformers for Environmental Sound Classification at the Edge
arXiv:2103.12157
Abstract
With the growth of the Internet of Things and the rise of Big Data, data processing and machine learning applications are being moved to cheap and low size, weight, and power (SWaP) devices at the edge, often in the form of mobile phones, embedded systems, or microcontrollers. The field of Cyber-Physical Measurements and Signature Intelligence (MASINT) makes use of these devices to analyze and exploit data in ways not otherwise possible, which results in increased data quality, increased security, and decreased bandwidth. However, methods to train and deploy models at the edge are limited, and models with sufficient accuracy are often too large for the edge device. Therefore, there is a clear need for techniques to create efficient AI/ML at the edge. This work presents training techniques for audio models in the field of environmental sound classification at the edge. Specifically, we design and train Transformers to classify office sounds in audio clips. Results show that a BERT-based Transformer, trained on Mel spectrograms, can outperform a CNN using 99.85% fewer parameters. To achieve this result, we first tested several audio feature extraction techniques designed for Transformers, using ESC-50 for evaluation, along with various augmentations. Our final model outperforms the state-of-the-art MFCC-based CNN on the office sounds dataset, using just over 6,000 parameters -- small enough to run on a microcontroller.
12 pages, submitted to IEEE Journal of Internet of Things
References in corpus (13)
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
- Language Models are Few-Shot Learners
- Neural Discrete Representation Learning
- Well-Read Students Learn Better: On the Importance of Pre-training Compact Models
- MCUNet: Tiny Deep Learning on IoT Devices
- Big Bird: Transformers for Longer Sequences
- Jukebox: A Generative Model for Music
- MusiCoder: A Universal Music-Acoustic Encoder Based on Transformers
- Transformer based unsupervised pre-training for acoustic representation learning
- Weak-Attention Suppression For Transformer Based Speech Recognition
- Translate Reverberated Speech to Anechoic Ones: Speech Dereverberation with BERT