Knowledge as a Teacher: Knowledge-Guided Structural Attention Networks

Abstract

Natural language understanding (NLU) is a core component of a spoken dialogue system. Recently recurrent neural networks (RNN) obtained strong results on NLU due to their superior ability of preserving sequential information over time. Traditionally, the NLU module tags semantic slots for utterances considering their flat structures, as the underlying RNN structure is a linear chain. However, natural language exhibits linguistic properties that provide rich, structured information for better understanding. This paper introduces a novel model, knowledge-guided structural attention networks (K-SAN), a generalization of RNN to additionally incorporate non-flat network topologies guided by prior knowledge. There are two characteristics: 1) important substructures can be captured from small training data, allowing the model to generalize to previously unseen test data; 2) the model automatically figures out the salient substructures that are essential to predict the semantic tags of the given sentences, so that the understanding performance can be improved. The experiments on the benchmark Air Travel Information System (ATIS) data show that the proposed K-SAN architecture can effectively extract salient knowledge from substructures with an attention mechanism, and outperform the performance of the state-of-the-art neural network based frameworks.

11 pages, 5 figures

References in corpus (7)

• Adam: A Method for Stochastic Optimization
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• On the Properties of Neural Machine Translation: Encoder-Decoder Approaches
• Convolutional Neural Networks for Sentence Classification
• Dynamic Memory Networks for Visual and Textual Question Answering
• Finding Function in Form: Compositional Character Models for Open Vocabulary Word Representation
• Dependency-based Convolutional Neural Networks for Sentence Embedding

Cited by in corpus (10)

• Neural Natural Language Inference Models Enhanced with External Knowledge
• Using Prior Knowledge to Guide BERT's Attention in Semantic Textual Matching Tasks
• Does injecting linguistic structure into language models lead to better alignment with brain recordings?
• Towards Unsupervised Language Understanding and Generation by Joint Dual Learning
• Dynamic Time-Aware Attention to Speaker Roles and Contexts for Spoken Language Understanding
• Real-time On-Demand Crowd-powered Entity Extraction
• End-to-End Joint Learning of Natural Language Understanding and Dialogue Manager
• Dynamically Context-Sensitive Time-Decay Attention for Dialogue Modeling
• Query-Variant Advertisement Text Generation with Association Knowledge
• Joint Learning of Word and Label Embeddings for Sequence Labelling in Spoken Language Understanding