Abstractive Text Summarization Using the BRIO Training Paradigm

Abstract

Summary sentences produced by abstractive summarization models may be coherent and comprehensive, but they lack control and rely heavily on reference summaries. The BRIO training paradigm assumes a non-deterministic distribution to reduce the model's dependence on reference summaries, and improve model performance during inference. This paper presents a straightforward but effective technique to improve abstractive summaries by fine-tuning pre-trained language models, and training them with the BRIO paradigm. We build a text summarization dataset for Vietnamese, called VieSum. We perform experiments with abstractive summarization models trained with the BRIO paradigm on the CNNDM and the VieSum datasets. The results show that the models, trained on basic hardware, outperform all existing abstractive summarization models, especially for Vietnamese.

arXiv comment6 pages, Findings of the Association for Computational Linguistics: ACL 2023

References in corpus (9)

• Longformer: The Long-Document Transformer
• mT5: A massively multilingual pre-trained text-to-text transformer
• Automatic Text Summarization of COVID-19 Medical Research Articles using BERT and GPT-2
• Actor-Critic based Training Framework for Abstractive Summarization
• Hierarchical Learning for Generation with Long Source Sequences
• Leveraging ParsBERT and Pretrained mT5 for Persian Abstractive Text Summarization
• Named Entity Inclusion in Abstractive Text Summarization
• BARTpho: Pre-trained Sequence-to-Sequence Models for Vietnamese
• Leveraging Locality in Abstractive Text Summarization