Analyzing Hidden Representations in End-to-End Automatic Speech Recognition Systems

Abstract

Neural models have become ubiquitous in automatic speech recognition systems. While neural networks are typically used as acoustic models in more complex systems, recent studies have explored end-to-end speech recognition systems based on neural networks, which can be trained to directly predict text from input acoustic features. Although such systems are conceptually elegant and simpler than traditional systems, it is less obvious how to interpret the trained models. In this work, we analyze the speech representations learned by a deep end-to-end model that is based on convolutional and recurrent layers, and trained with a connectionist temporal classification (CTC) loss. We use a pre-trained model to generate frame-level features which are given to a classifier that is trained on frame classification into phones. We evaluate representations from different layers of the deep model and compare their quality for predicting phone labels. Our experiments shed light on important aspects of the end-to-end model such as layer depth, model complexity, and other design choices.

NIPS 2017

References in corpus (6)

• Adam: A Method for Stochastic Optimization
• End-to-end Continuous Speech Recognition using Attention-based Recurrent NN: First Results
• What do Neural Machine Translation Models Learn about Morphology?
• EESEN: End-to-End Speech Recognition using Deep RNN Models and WFST-based Decoding
• Representations of language in a model of visually grounded speech signal
• From phonemes to images: levels of representation in a recurrent neural model of visually-grounded language learning

Cited by in corpus (6)

• Visually grounded models of spoken language: A survey of datasets, architectures and evaluation techniques
• Analyzing analytical methods: The case of phonology in neural models of spoken language
• Probing Neural Dialog Models for Conversational Understanding
• Wave to Syntax: Probing spoken language models for syntax
• Interpreting intermediate convolutional layers in unsupervised acoustic word classification
• MixRep: Hidden Representation Mixup for Low-Resource Speech Recognition