End-to-End ASR-free Keyword Search from Speech

Abstract

End-to-end (E2E) systems have achieved competitive results compared to conventional hybrid hidden Markov model (HMM)-deep neural network based automatic speech recognition (ASR) systems. Such E2E systems are attractive due to the lack of dependence on alignments between input acoustic and output grapheme or HMM state sequence during training. This paper explores the design of an ASR-free end-to-end system for text query-based keyword search (KWS) from speech trained with minimal supervision. Our E2E KWS system consists of three sub-systems. The first sub-system is a recurrent neural network (RNN)-based acoustic auto-encoder trained to reconstruct the audio through a finite-dimensional representation. The second sub-system is a character-level RNN language model using embeddings learned from a convolutional neural network. Since the acoustic and text query embeddings occupy different representation spaces, they are input to a third feed-forward neural network that predicts whether the query occurs in the acoustic utterance or not. This E2E ASR-free KWS system performs respectably despite lacking a conventional ASR system and trains much faster.

arXiv commentPublished in the IEEE 2017 International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2017), scheduled for 5-9 March 2017 in New Orleans, Louisiana, USA

References in corpus (7)

• Adam: A Method for Stochastic Optimization
• Neural Machine Translation by Jointly Learning to Align and Translate
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Character-Aware Neural Language Models
• The Microsoft 2016 Conversational Speech Recognition System
• EESEN: End-to-End Speech Recognition using Deep RNN Models and WFST-based Decoding
• Multi-view Recurrent Neural Acoustic Word Embeddings

Cited by in corpus (18)

• Unsupervised Automatic Speech Recognition: A Review
• Direct Speech-to-image Translation
• Seeing wake words: Audio-visual Keyword Spotting
• Sequence Discriminative Training for Deep Learning based Acoustic Keyword Spotting
• An Investigation of Few-Shot Learning in Spoken Term Classification
• Fast End-to-End Speech Recognition via Non-Autoregressive Models and Cross-Modal Knowledge Transferring from BERT
• Small-Footprint Open-Vocabulary Keyword Spotting with Quantized LSTM Networks
• Visual Keyword Spotting with Attention
• End-to-End Open Vocabulary Keyword Search With Multilingual Neural Representations
• Query-by-example Spoken Term Detection using Attention-based Multi-hop Networks
• Multilingual Jointly Trained Acoustic and Written Word Embeddings
• Query-by-example on-device keyword spotting
• Automatic Detection of Phonological Errors in Child Speech Using Siamese Recurrent Autoencoder
• Acoustically Grounded Word Embeddings for Improved Acoustics-to-Word Speech Recognition
• Semantic query-by-example speech search using visual grounding
• A multi-view approach for Mandarin non-native mispronunciation verification
• LSTM Acoustic Models Learn to Align and Pronounce with Graphemes
• Overlapped speech recognition from a jointly learned multi-channel neural speech extraction and representation