Analyzing Multi-Task Learning for Abstractive Text Summarization

Abstract

Despite the recent success of multi-task learning and pre-finetuning for natural language understanding, few works have studied the effects of task families on abstractive text summarization. Task families are a form of task grouping during the pre-finetuning stage to learn common skills, such as reading comprehension. To close this gap, we analyze the influence of multi-task learning strategies using task families for the English abstractive text summarization task. We group tasks into one of three strategies, i.e., sequential, simultaneous, and continual multi-task learning, and evaluate trained models through two downstream tasks. We find that certain combinations of task families (e.g., advanced reading comprehension and natural language inference) positively impact downstream performance. Further, we find that choice and combinations of task families influence downstream performance more than the training scheme, supporting the use of task families for abstractive text summarization.

References in corpus (18)

• Overcoming catastrophic forgetting in neural networks
• BERTScore: Evaluating Text Generation with BERT
• SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
• PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
• Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales
• CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge
• ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehension
• Continual Lifelong Learning in Natural Language Processing: A Survey
• WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations
• Neural Media Bias Detection Using Distant Supervision With BABE -- Bias Annotations By Experts
• The Factual Inconsistency Problem in Abstractive Text Summarization: A Survey
• CTRLsum: Towards Generic Controllable Text Summarization
• Identifying Machine-Paraphrased Plagiarism
• Quick and (not so) Dirty: Unsupervised Selection of Justification Sentences for Multi-hop Question Answering
• Exploiting Transformer-based Multitask Learning for the Detection of Media Bias in News Articles
• Semantics-aware BERT for Language Understanding
• Testing the Generalization of Neural Language Models for COVID-19 Misinformation Detection
• CommonGen: A Constrained Text Generation Challenge for Generative Commonsense Reasoning