Feedforward Sequential Memory Neural Networks without Recurrent Feedback

Abstract

We introduce a new structure for memory neural networks, called feedforward sequential memory networks (FSMN), which can learn long-term dependency without using recurrent feedback. The proposed FSMN is a standard feedforward neural networks equipped with learnable sequential memory blocks in the hidden layers. In this work, we have applied FSMN to several language modeling (LM) tasks. Experimental results have shown that the memory blocks in FSMN can learn effective representations of long history. Experiments have shown that FSMN based language models can significantly outperform not only feedforward neural network (FNN) based LMs but also the popular recurrent neural network (RNN) LMs.

4 pages, 1figure

References in corpus (3)

• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Large-scale Simple Question Answering with Memory Networks
• Learning Longer Memory in Recurrent Neural Networks

Cited by in corpus (7)

• Transformer-based Acoustic Modeling for Hybrid Speech Recognition
• Feedforward Sequential Memory Networks: A New Structure to Learn Long-term Dependency
• Deformable TDNN with adaptive receptive fields for speech recognition
• Faster, Simpler and More Accurate Hybrid ASR Systems Using Wordpieces
• Deep Feed-forward Sequential Memory Networks for Speech Synthesis
• Transformer in action: a comparative study of transformer-based acoustic models for large scale speech recognition applications
• On Training Bi-directional Neural Network Language Model with Noise Contrastive Estimation