HAN: Higher-order Attention Network for Spoken Language Understanding

Abstract

Spoken Language Understanding (SLU), including intent detection and slot filling, is a core component in human-computer interaction. The natural attributes of the relationship among the two subtasks make higher requirements on fine-grained feature interaction, i.e., the token-level intent features and slot features. Previous works mainly focus on jointly modeling the relationship between the two subtasks with attention-based models, while ignoring the exploration of attention order. In this paper, we propose to replace the conventional attention with our proposed Bilinear attention block and show that the introduced Higher-order Attention Network (HAN) brings improvement for the SLU task. Importantly, we conduct wide analysis to explore the effectiveness brought from the higher-order attention.

References in corpus (9)

• Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces
• Continuously Differentiable Exponential Linear Units
• Hierarchically-Refined Label Attention Network for Sequence Labeling
• A Novel Bi-directional Interrelated Model for Joint Intent Detection and Slot Filling
• A Co-Interactive Transformer for Joint Slot Filling and Intent Detection
• CM-Net: A Novel Collaborative Memory Network for Spoken Language Understanding
• PIN: A Novel Parallel Interactive Network for Spoken Language Understanding
• Injecting Word Information with Multi-Level Word Adapter for Chinese Spoken Language Understanding
• A Stack-Propagation Framework with Token-Level Intent Detection for Spoken Language Understanding