A New Sentence Ordering Method Using BERT Pretrained Model

Abstract

Building systems with capability of natural language understanding (NLU) has been one of the oldest areas of AI. An essential component of NLU is to detect logical succession of events contained in a text. The task of sentence ordering is proposed to learn succession of events with applications in AI tasks. The performance of previous works employing statistical methods is poor, while the neural networks-based approaches are in serious need of large corpora for model learning. In this paper, we propose a method for sentence ordering which does not need a training phase and consequently a large corpus for learning. To this end, we generate sentence embedding using BERT pre-trained model and measure sentence similarity using cosine similarity score. We suggest this score as an indicator of sequential events' level of coherence. We finally sort the sentences through brute-force search to maximize overall similarities of the sequenced sentences. Our proposed method outperformed other baselines on ROCStories, a corpus of 5-sentence human-made stories. The method is specifically more efficient than neural network-based methods when no huge corpus is available. Among other advantages of this method are its interpretability and needlessness to linguistic knowledge.

arXiv comment7 pages, 4 figures, 2020 11th International Conference on Information and Knowledge Technology (IKT)

References in corpus (15)

• Sequence to Sequence Learning with Neural Networks
• Universal Sentence Encoder
• Skip-Thought Vectors
• QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension
• Inferring Strategies for Sentence Ordering in Multidocument News Summarization
• Catching the Drift: Probabilistic Content Models, with Applications to Generation and Summarization
• End-to-End Neural Sentence Ordering Using Pointer Network
• Neural Sentence Ordering
• Neural Net Models for Open-Domain Discourse Coherence
• SBERT-WK: A Sentence Embedding Method by Dissecting BERT-based Word Models
• Graph-based Neural Sentence Ordering
• Correlation Coefficients and Semantic Textual Similarity
• Combining Word Embeddings and N-grams for Unsupervised Document Summarization
• Topological Sort for Sentence Ordering
• Learning Sentence Embeddings for Coherence Modelling and Beyond