Match-Ignition: Plugging PageRank into Transformer for Long-form Text Matching
arXiv:2101.06423 · doi:10.1145/3459637.3482450
Abstract
Neural text matching models have been widely used in community question answering, information retrieval, and dialogue. However, these models designed for short texts cannot well address the long-form text matching problem, because there are many contexts in long-form texts can not be directly aligned with each other, and it is difficult for existing models to capture the key matching signals from such noisy data. Besides, these models are computationally expensive for simply use all textual data indiscriminately. To tackle the effectiveness and efficiency problem, we propose a novel hierarchical noise filtering model, namely Match-Ignition. The main idea is to plug the well-known PageRank algorithm into the Transformer, to identify and filter both sentence and word level noisy information in the matching process. Noisy sentences are usually easy to detect because previous work has shown that their similarity can be explicitly evaluated by the word overlapping, so we directly use PageRank to filter such information based on a sentence similarity graph. Unlike sentences, words rely on their contexts to express concrete meanings, so we propose to jointly learn the filtering and matching process, to well capture the critical word-level matching signals. Specifically, a word graph is first built based on the attention scores in each self-attention block of Transformer, and key words are then selected by applying PageRank on this graph. In this way, noisy words will be filtered out layer by layer in the matching process. Experimental results show that Match-Ignition outperforms both SOTA short text matching models and recent long-form text matching models. We also conduct detailed analysis to show that Match-Ignition efficiently captures important sentences and words, to facilitate the long-form text matching process.
arXiv commentAccepted as long paper by CIKM 2021
References in corpus (18)
- Adam: A Method for Stochastic Optimization
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- TensorFlow: A system for large-scale machine learning
- Longformer: The Long-Document Transformer
- Convolutional Neural Network Architectures for Matching Natural Language Sentences
- A Deep Relevance Matching Model for Ad-hoc Retrieval
- Deep Sentence Embedding Using Long Short-Term Memory Networks: Analysis and Application to Information Retrieval
- DeepRank: A New Deep Architecture for Relevance Ranking in Information Retrieval
- Long Range Arena: A Benchmark for Efficient Transformers
- A Deep Architecture for Semantic Matching with Multiple Positional Sentence Representations
- Match-SRNN: Modeling the Recursive Matching Structure with Spatial RNN
- Bilateral Multi-Perspective Matching for Natural Language Sentences
- Modeling Diverse Relevance Patterns in Ad-hoc Retrieval
- Beyond 512 Tokens: Siamese Multi-depth Transformer-based Hierarchical Encoder for Long-Form Document Matching
- A Study of MatchPyramid Models on Ad-hoc Retrieval
- On Identifiability in Transformers
- MatchZoo: A Toolkit for Deep Text Matching
- Simple and Effective Text Matching with Richer Alignment Features