LadRa-Net: Locally-Aware Dynamic Re-read Attention Net for Sentence Semantic Matching

Abstract

Sentence semantic matching requires an agent to determine the semantic relation between two sentences, which is widely used in various natural language tasks, such as Natural Language Inference (NLI), Paraphrase Identification (PI), and so on. Much recent progress has been made in this area, especially attention-based methods and pre-trained language model based methods. However, most of these methods focus on all the important parts in sentences in a static way and only emphasize how important the words are to the query, inhibiting the ability of attention mechanism. In order to overcome this problem and boost the performance of attention mechanism, we propose a novel dynamic re-read attention, which can pay close attention to one small region of sentences at each step and re-read the important parts for better sentence representations. Based on this attention variation, we develop a novel Dynamic Re-read Network (DRr-Net) for sentence semantic matching. Moreover, selecting one small region in dynamic re-read attention seems insufficient for sentence semantics, and employing pre-trained language models as input encoders will introduce incomplete and fragile representation problems. To this end, we extend DRrNet to Locally-Aware Dynamic Re-read Attention Net (LadRa-Net), in which local structure of sentences is employed to alleviate the shortcoming of Byte-Pair Encoding (BPE) in pre-trained language models and boost the performance of dynamic reread attention. Extensive experiments on two popular sentence semantic matching tasks demonstrate that DRr-Net can significantly improve the performance of sentence semantic matching. Meanwhile, LadRa-Net is able to achieve better performance by considering the local structures of sentences. In addition, it is exceedingly interesting that some discoveries in our experiments are consistent with some findings of psychological research.

arXiv commentAccpeted by IEEE Transactions on Neural Networks and Learning Systems

References in corpus (16)

• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Enhanced LSTM for Natural Language Inference
• Convolutional Neural Networks for Sentence Classification
• ERNIE: Enhanced Representation through Knowledge Integration
• Describing Multimedia Content using Attention-based Encoder--Decoder Networks
• Learning Natural Language Inference using Bidirectional LSTM model and Inner-Attention
• Natural Language Inference over Interaction Space
• Sentence Similarity Learning by Lexical Decomposition and Composition
• Modeling Coverage for Neural Machine Translation
• Bilateral Multi-Perspective Matching for Natural Language Sentences
• Neural Network Models for Paraphrase Identification, Semantic Textual Similarity, Natural Language Inference, and Question Answering
• CharBERT: Character-aware Pre-trained Language Model
• A Hierarchical Attention Model for Social Contextual Image Recommendation
• Distance-based Self-Attention Network for Natural Language Inference
• Dynamic Self-Attention : Computing Attention over Words Dynamically for Sentence Embedding
• A Survey on Curriculum Learning

Cited by in corpus (1)

• Information Retrieval: Recent Advances and Beyond