Gram-CTC: Automatic Unit Selection and Target Decomposition for Sequence Labelling

Abstract

Most existing sequence labelling models rely on a fixed decomposition of a target sequence into a sequence of basic units. These methods suffer from two major drawbacks: 1) the set of basic units is fixed, such as the set of words, characters or phonemes in speech recognition, and 2) the decomposition of target sequences is fixed. These drawbacks usually result in sub-optimal performance of modeling sequences. In this pa- per, we extend the popular CTC loss criterion to alleviate these limitations, and propose a new loss function called Gram-CTC. While preserving the advantages of CTC, Gram-CTC automatically learns the best set of basic units (grams), as well as the most suitable decomposition of tar- get sequences. Unlike CTC, Gram-CTC allows the model to output variable number of characters at each time step, which enables the model to capture longer term dependency and improves the computational efficiency. We demonstrate that the proposed Gram-CTC improves CTC in terms of both performance and efficiency on the large vocabulary speech recognition task at multiple scales of data, and that with Gram-CTC we can outperform the state-of-the-art on a standard speech benchmark.

arXiv commentPublished at ICML 2017

References in corpus (14)

• Sequence to Sequence Learning with Neural Networks
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
• Achieving Human Parity in Conversational Speech Recognition
• Hierarchical Multiscale Recurrent Neural Networks
• Wav2Letter: an End-to-End ConvNet-based Speech Recognition System
• Sequence-Level Knowledge Distillation
• Advances in All-Neural Speech Recognition
• Towards better decoding and language model integration in sequence to sequence models
• Neural Speech Recognizer: Acoustic-to-Word LSTM Model for Large Vocabulary Speech Recognition
• Dense Prediction on Sequences with Time-Dilated Convolutions for Speech Recognition
• Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learning
• Very Deep Convolutional Networks for End-to-End Speech Recognition

Cited by in corpus (15)

• Improved training of end-to-end attention models for speech recognition
• Letter-Based Speech Recognition with Gated ConvNets
• Imputer: Sequence Modelling via Imputation and Dynamic Programming
• Reducing Bias in Production Speech Models
• Acoustic Data-Driven Subword Modeling for End-to-End Speech Recognition
• Advancing Acoustic-to-Word CTC Model
• Multiscale sequence modeling with a learned dictionary
• Trace norm regularization and faster inference for embedded speech recognition RNNs
• Optimal Completion Distillation for Sequence Learning
• Advancing Connectionist Temporal Classification With Attention Modeling
• On the Inductive Bias of Word-Character-Level Multi-Task Learning for Speech Recognition
• Hybrid CTC-Attention based End-to-End Speech Recognition using Subword Units
• Recent Progresses in Deep Learning based Acoustic Models (Updated)
• Advancing Acoustic-to-Word CTC Model with Attention and Mixed-Units
• Large Margin Neural Language Model