Neural Natural Language Processing for Long Texts: A Survey on Classification and Summarization

Abstract

The adoption of Deep Neural Networks (DNNs) has greatly benefited Natural Language Processing (NLP) during the past decade. However, the demands of long document analysis are quite different from those of shorter texts, while the ever increasing size of documents uploaded online renders automated understanding of lengthy texts a critical issue. Relevant applications include automated Web mining, legal document review, medical records analysis, financial reports analysis, contract management, environmental impact assessment, news aggregation, etc. Despite the relatively recent development of efficient algorithms for analyzing long documents, practical tools in this field are currently flourishing. This article serves as an entry point into this dynamic domain and aims to achieve two objectives. First of all, it provides an introductory overview of the relevant neural building blocks, serving as a concise tutorial for the field. Secondly, it offers a brief examination of the current state-of-the-art in two key long document analysis tasks: document classification and document summarization. Sentiment analysis for long texts is also covered, since it is typically treated as a particular case of document classification. Consequently, this article presents an introductory exploration of document-level analysis, addressing the primary challenges, concerns, and existing solutions. Finally, it offers a concise definition of "long text/document", presents an original overarching taxonomy of common deep neural methods for long document analysis and lists publicly available annotated datasets that can facilitate further research in this area.

arXiv comment65 pages, 11 figures, 5 tables

References in corpus (28)

• Efficient Estimation of Word Representations in Vector Space
• Neural Machine Translation by Jointly Learning to Align and Translate
• BioBERT: a pre-trained biomedical language representation model for biomedical text mining
• LLaMA: Open and Efficient Foundation Language Models
• Longformer: The Long-Document Transformer
• PaLM: Scaling Language Modeling with Pathways
• Text Classification Algorithms: A Survey
• Universal Sentence Encoder
• Scaling Instruction-Finetuned Language Models
• Gemini: A Family of Highly Capable Multimodal Models
• Training Compute-Optimal Large Language Models
• BARTScore: Evaluating Generated Text as Text Generation
• Natural Language Processing Advancements By Deep Learning: A Survey
• News Summarization and Evaluation in the Era of GPT-3
• Data Augmentation in Natural Language Processing: A Novel Text Generation Approach for Long and Short Text Classifiers
• A Practical Survey on Faster and Lighter Transformers
• An Empirical Survey on Long Document Summarization: Datasets, Models and Metrics
• OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization
• A Survey of Active Learning for Text Classification using Deep Neural Networks
• Conceptualized Representation Learning for Chinese Biomedical Text Mining
• End-to-End Neural Sentence Ordering Using Pointer Network
• Long-length Legal Document Classification
• The Power of Selecting Key Blocks with Local Pre-ranking for Long Document Information Retrieval
• LongForm: Effective Instruction Tuning with Reverse Instructions
• Contextualized Topic Coherence Metrics
• Urdu Speech and Text Based Sentiment Analyzer
• ERNIE-SPARSE: Learning Hierarchical Efficient Transformer Through Regularized Self-Attention
• Computational Storytelling and Emotions: A Survey