Longformer: The Long-Document Transformer

Abstract

Transformer-based models are unable to process long sequences due to their self-attention operation, which scales quadratically with the sequence length. To address this limitation, we introduce the Longformer with an attention mechanism that scales linearly with sequence length, making it easy to process documents of thousands of tokens or longer. Longformer's attention mechanism is a drop-in replacement for the standard self-attention and combines a local windowed attention with a task motivated global attention. Following prior work on long-sequence transformers, we evaluate Longformer on character-level language modeling and achieve state-of-the-art results on text8 and enwik8. In contrast to most prior work, we also pretrain Longformer and finetune it on a variety of downstream tasks. Our pretrained Longformer consistently outperforms RoBERTa on long document tasks and sets new state-of-the-art results on WikiHop and TriviaQA. We finally introduce the Longformer-Encoder-Decoder (LED), a Longformer variant for supporting long document generative sequence-to-sequence tasks, and demonstrate its effectiveness on the arXiv summarization dataset.

Version 2 introduces the Longformer-Encoder-Decoder (LED) model

References in corpus (15)

• Sequence to Sequence Learning with Neural Networks
• WaveNet: A Generative Model for Raw Audio
• Unsupervised Data Augmentation for Consistency Training
• Training Deep Nets with Sublinear Memory Cost
• Generating Long Sequences with Sparse Transformers
• Semi-supervised Sequence Learning
• Reformer: The Efficient Transformer
• A Simple Method for Commonsense Reasoning
• Big Bird: Transformers for Longer Sequences
• fairseq: A Fast, Extensible Toolkit for Sequence Modeling
• Multi-hop Question Answering via Reasoning Chains
• BP-Transformer: Modelling Long-Range Context via Binary Partitioning
• GMAT: Global Memory Augmentation for Transformers
• Select, Answer and Explain: Interpretable Multi-hop Reading Comprehension over Multiple Documents
• Graph Sequential Network for Reasoning over Sequences

Cited by in corpus (367)

• Survey of Hallucination in Natural Language Generation
• Transformers in Vision: A Survey
• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• On the Opportunities and Risks of Foundation Models
• Deformable DETR: Deformable Transformers for End-to-End Object Detection
• Linformer: Self-Attention with Linear Complexity
• Human Action Recognition from Various Data Modalities: A Review
• Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting
• DeBERTa: Decoding-enhanced BERT with Disentangled Attention
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
• MetaFormer Baselines for Vision
• Dual Aspect Self-Attention based on Transformer for Remaining Useful Life Prediction
• Big Bird: Transformers for Longer Sequences
• Focal Self-attention for Local-Global Interactions in Vision Transformers
• TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Up
• XCiT: Cross-Covariance Image Transformers
• Spatio-Temporal Wind Speed Forecasting using Graph Networks and Novel Transformer Architectures
• Perceiver IO: A General Architecture for Structured Inputs & Outputs
• You Only Look at One Sequence: Rethinking Transformer in Vision through Object Detection
• Long Range Arena: A Benchmark for Efficient Transformers
• Video Transformers: A Survey
• The NLP Cookbook: Modern Recipes for Transformer based Deep Learning Architectures
• Differentiable Quantum Architecture Search
• A Comparative Study of Pretrained Language Models for Long Clinical Text
• Transformers and Large Language Models for Efficient Intrusion Detection Systems: A Comprehensive Survey
• Enable Deep Learning on Mobile Devices: Methods, Systems, and Applications
• A Practical Survey on Faster and Lighter Transformers
• Making Pre-trained Language Models Better Few-shot Learners
• CenterCLIP: Token Clustering for Efficient Text-Video Retrieval
• Perceiver: General Perception with Iterative Attention
• Shuffle Transformer: Rethinking Spatial Shuffle for Vision Transformer
• Random Feature Attention
• Rethinking Attention with Performers
• Supervised and Unsupervised Neural Approaches to Text Readability
• An Empirical Survey on Long Document Summarization: Datasets, Models and Metrics
• Fredformer: Frequency Debiased Transformer for Time Series Forecasting
• Pretrained Transformers for Text Ranking: BERT and Beyond
• Advances of Machine Learning in Materials Science: Ideas and Techniques
• Local-Global Context Aware Transformer for Language-Guided Video Segmentation
• Materials science in the era of large language models: a perspective
• CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windows
• Aligning AI With Shared Human Values
• Pretrained Transformers as Universal Computation Engines
• Rethinking Search: Making Domain Experts out of Dilettantes
• Word-Level Coreference Resolution
• Chasing Sparsity in Vision Transformers: An End-to-End Exploration
• Does the Magic of BERT Apply to Medical Code Assignment? A Quantitative Study
• Fastformer: Additive Attention Can Be All You Need
• Annotating Columns with Pre-trained Language Models
• Full Page Handwriting Recognition via Image to Sequence Extraction
• Towards Accurate Post-Training Quantization for Vision Transformer
• NeurIPS 2020 EfficientQA Competition: Systems, Analyses and Lessons Learned
• Neural Natural Language Processing for Long Texts: A Survey on Classification and Summarization
• ConvTransformer: A Convolutional Transformer Network for Video Frame Synthesis
• Exploring the Synergies of Hybrid CNNs and ViTs Architectures for Computer Vision: A survey
• Space-time Mixing Attention for Video Transformer
• VST++: Efficient and Stronger Visual Saliency Transformer
• Long-Short Transformer: Efficient Transformers for Language and Vision
• Discharge Summary Hospital Course Summarisation of In Patient Electronic Health Record Text with Clinical Concept Guided Deep Pre-Trained Transformer Models
• Luna: Linear Unified Nested Attention
• Hierarchical Label-wise Attention Transformer Model for Explainable ICD Coding
• Keeping Your Eye on the Ball: Trajectory Attention in Video Transformers
• LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domain
• LambdaNetworks: Modeling Long-Range Interactions Without Attention
• DeLighT: Deep and Light-weight Transformer
• Few-Shot Bot: Prompt-Based Learning for Dialogue Systems
• Self-Attention Between Datapoints: Going Beyond Individual Input-Output Pairs in Deep Learning
• GraphiT: Encoding Graph Structure in Transformers
• Hierarchical Learning for Generation with Long Source Sequences
• Identifying Machine-Paraphrased Plagiarism
• AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing
• A Unified Review of Deep Learning for Automated Medical Coding
• A Dataset for Answering Time-Sensitive Questions
• Language Models as Few-Shot Learner for Task-Oriented Dialogue Systems
• Mokey: Enabling Narrow Fixed-Point Inference for Out-of-the-Box Floating-Point Transformer Models
• Multimodal Fusion and Vision-Language Models: A Survey for Robot Vision
• Generation-Augmented Retrieval for Open-domain Question Answering
• SODA: A Natural Language Processing Package to Extract Social Determinants of Health for Cancer Studies
• MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers
• HTLM: Hyper-Text Pre-Training and Prompting of Language Models
• Are Neural Language Models Good Plagiarists? A Benchmark for Neural Paraphrase Detection
• Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation
• "This is Fake! Shared it by Mistake": Assessing the Intent of Fake News Spreaders
• Hierarchical Graph Network for Multi-hop Question Answering
• Nyströmformer: A Nyström-Based Algorithm for Approximating Self-Attention
• Masked Language Modeling for Proteins via Linearly Scalable Long-Context Transformers
• Text Guide: Improving the quality of long text classification by a text selection method based on feature importance
• How Large Language Models are Transforming Machine-Paraphrased Plagiarism
• Domain-specific ChatBots for Science using Embeddings
• ETC: Encoding Long and Structured Inputs in Transformers
• Adaptive Fourier Neural Operators: Efficient Token Mixers for Transformers
• A Survey of Deep Learning Approaches for OCR and Document Understanding
• Efficient Quantized Sparse Matrix Operations on Tensor Cores
• GTrans: Grouping and Fusing Transformer Layers for Neural Machine Translation
• ConceptEVA: Concept-Based Interactive Exploration and Customization of Document Summaries
• ZeroBERTo: Leveraging Zero-Shot Text Classification by Topic Modeling
• TVT: Transferable Vision Transformer for Unsupervised Domain Adaptation
• Mind the Gap: Assessing Temporal Generalization in Neural Language Models
• Vulgar Remarks Detection in Chittagonian Dialect of Bangla
• Detection of depression on social networks using transformers and ensembles
• XAI in Computational Linguistics: Understanding Political Leanings in the Slovenian Parliament
• Language-Independent Sentiment Labelling with Distant Supervision: A Case Study for English, Sepedi and Setswana
• From Statistical Methods to Deep Learning, Automatic Keyphrase Prediction: A Survey
• Pile-Up Mitigation using Attention
• GMAT: Global Memory Augmentation for Transformers
• Centroid Transformers: Learning to Abstract with Attention
• Match-Ignition: Plugging PageRank into Transformer for Long-form Text Matching
• RealFormer: Transformer Likes Residual Attention
• DeTiME: Diffusion-Enhanced Topic Modeling using Encoder-decoder based LLM
• Summ^N: A Multi-Stage Summarization Framework for Long Input Dialogues and Documents
• Detecting Hallucinated Content in Conditional Neural Sequence Generation
• Homogeneous Tokenizer Matters: Homogeneous Visual Tokenizer for Remote Sensing Image Understanding
• GNN-LM: Language Modeling based on Global Contexts via GNN
• SYMBA: Symbolic Computation of Squared Amplitudes in High Energy Physics with Machine Learning
• Transformers: "The End of History" for NLP?
• Answering Complex Open-Domain Questions with Multi-Hop Dense Retrieval
• Cluster-Former: Clustering-based Sparse Transformer for Long-Range Dependency Encoding
• Conformer-Kernel with Query Term Independence for Document Retrieval
• Generating a Structured Summary of Numerous Academic Papers: Dataset and Method
• Multitask Balanced and Recalibrated Network for Medical Code Prediction
• Efficient Transformers with Dynamic Token Pooling
• Coordination Among Neural Modules Through a Shared Global Workspace
• Multi-scale Time-stepping of Partial Differential Equations with Transformers
• Self-Supervised Document Similarity Ranking via Contextualized Language Models and Hierarchical Inference
• Blockwise Self-Attention for Long Document Understanding
• Bridging Textual and Tabular Data for Cross-Domain Text-to-SQL Semantic Parsing
• Pre-Trained Models: Past, Present and Future
• The Power of Selecting Key Blocks with Local Pre-ranking for Long Document Information Retrieval
• A Survey of Text Representation Methods and Their Genealogy
• Lawformer: A Pre-trained Language Model for Chinese Legal Long Documents
• Towards Clinical Encounter Summarization: Learning to Compose Discharge Summaries from Prior Notes
• Learning To Retrieve: How to Train a Dense Retrieval Model Effectively and Efficiently
• Pre-Trained Language Models for Keyphrase Prediction: A Review
• A Scoping Review of Natural Language Processing in Addressing Medically Inaccurate Information: Errors, Misinformation, and Hallucination
• Hi-BEHRT: Hierarchical Transformer-based model for accurate prediction of clinical events using multimodal longitudinal electronic health records
• Fast Convergence of DETR with Spatially Modulated Co-Attention
• Fast and Effective Biomedical Entity Linking Using a Dual Encoder
• Challenges in Domain-Specific Abstractive Summarization and How to Overcome them
• SparseBERT: Rethinking the Importance Analysis in Self-attention
• Transformer Acceleration with Dynamic Sparse Attention
• Graph-free Multi-hop Reading Comprehension: A Select-to-Guide Strategy
• The NLP Task Effectiveness of Long-Range Transformers
• DMDD: A Large-Scale Dataset for Dataset Mentions Detection
• Overview of the BioLaySumm 2023 Shared Task on Lay Summarization of Biomedical Research Articles
• A Meta-Evaluation of Faithfulness Metrics for Long-Form Hospital-Course Summarization
• Abstractive Text Summarization Using the BRIO Training Paradigm
• Learn To Remember: Transformer with Recurrent Memory for Document-Level Machine Translation
• SciCo: Hierarchical Cross-Document Coreference for Scientific Concepts
• Conflicts, Villains, Resolutions: Towards models of Narrative Media Framing
• Efficient Attentions for Long Document Summarization
• ERNIE-Doc: A Retrospective Long-Document Modeling Transformer
• Fourier Transformer: Fast Long Range Modeling by Removing Sequence Redundancy with FFT Operator
• Intra-Document Cascading: Learning to Select Passages for Neural Document Ranking
• SC4ANM: Identifying Optimal Section Combinations for Automated Novelty Prediction in Academic Papers
• SFTformer: A Spatial-Frequency-Temporal Correlation-Decoupling Transformer for Radar Echo Extrapolation
• Cura: Curation at Social Media Scale
• $O(n)$ Connections are Expressive Enough: Universal Approximability of Sparse Transformers
• LazyFormer: Self Attention with Lazy Update
• An Unbiased Transformer Source Code Learning with Semantic Vulnerability Graph
• Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search
• Towards Suicide Prevention from Bipolar Disorder with Temporal Symptom-Aware Multitask Learning
• Co-BERT: A Context-Aware BERT Retrieval Model Incorporating Local and Query-specific Context
• Personal Entity, Concept, and Named Entity Linking in Conversations
• Scatterbrain: Unifying Sparse and Low-rank Attention Approximation
• DynaPipe: Optimizing Multi-task Training through Dynamic Pipelines
• Gender Bias Detection in Court Decisions: A Brazilian Case Study
• Medical Knowledge Graph QA for Drug-Drug Interaction Prediction based on Multi-hop Machine Reading Comprehension
• MS2: Multi-Document Summarization of Medical Studies
• Obstacle-Transformer: A Trajectory Prediction Network Based on Surrounding Trajectories
• Towards a Multi-modal, Multi-task Learning based Pre-training Framework for Document Representation Learning
• Automated essay scoring using efficient transformer-based language models
• TR-BERT: Dynamic Token Reduction for Accelerating BERT Inference
• Explaining The Efficacy of Counterfactually Augmented Data
• SelfDoc: Self-Supervised Document Representation Learning
• End-to-End Attention-based Image Captioning
• Kleister: Key Information Extraction Datasets Involving Long Documents with Complex Layouts
• Malicious or Benign? Towards Effective Content Moderation for Children's Videos
• Multi-View Sequence-to-Sequence Models with Conversational Structure for Abstractive Dialogue Summarization
• LAMPRET: Layout-Aware Multimodal PreTraining for Document Understanding
• Transformers for End-to-End InfoSec Tasks: A Feasibility Study
• SkillSpan: Hard and Soft Skill Extraction from English Job Postings
• Analysis of Twitter Users' Lifestyle Choices using Joint Embedding Model
• A Two-Phase Approach for Abstractive Podcast Summarization
• Grid Search Hyperparameter Benchmarking of BERT, ALBERT, and LongFormer on DuoRC
• Analysis of the Evolution of Advanced Transformer-Based Language Models: Experiments on Opinion Mining
• Large Language Models as Autonomous Spacecraft Operators in Kerbal Space Program
• Learning the Simplicity of Scattering Amplitudes
• D-FaST: Cognitive Signal Decoding with Disentangled Frequency-Spatial-Temporal Attention
• Deep Neural Networks and Brain Alignment: Brain Encoding and Decoding (Survey)
• Connecting the Dots: What Graph-Based Text Representations Work Best for Text Classification Using Graph Neural Networks?
• Human Language Modeling
• On the Trade-off between Redundancy and Local Coherence in Summarization
• When FastText Pays Attention: Efficient Estimation of Word Representations using Constrained Positional Weighting
• SJTU-NICT's Supervised and Unsupervised Neural Machine Translation Systems for the WMT20 News Translation Task
• Summarize, Outline, and Elaborate: Long-Text Generation via Hierarchical Supervision from Extractive Summaries
• Question-Answering Approach to Evaluating Legal Summaries
• Eigen Analysis of Self-Attention and its Reconstruction from Partial Computation
• Document Classification for COVID-19 Literature
• Language Modeling using LMUs: 10x Better Data Efficiency or Improved Scaling Compared to Transformers
• Efficient Conformer with Prob-Sparse Attention Mechanism for End-to-EndSpeech Recognition
• A Multilingual Translator to SQL with Database Schema Pruning to Improve Self-Attention
• Longformer for MS MARCO Document Re-ranking Task
• Multi-document Summarization: A Comparative Evaluation
• Journalistic Guidelines Aware News Image Captioning
• Iterative Hierarchical Attention for Answering Complex Questions over Long Documents
• Embodied BERT: A Transformer Model for Embodied, Language-guided Visual Task Completion
• Investigating the Effects of Sparse Attention on Cross-Encoders
• FMMformer: Efficient and Flexible Transformer via Decomposed Near-field and Far-field Attention
• Not All Memories are Created Equal: Learning to Forget by Expiring
• Code Summarization with Structure-induced Transformer
• Which *BERT? A Survey Organizing Contextualized Encoders
• What Context Features Can Transformer Language Models Use?
• Adding Recurrence to Pretrained Transformers for Improved Efficiency and Context Size
• PoNet: Pooling Network for Efficient Token Mixing in Long Sequences
• A Unified Efficient Pyramid Transformer for Semantic Segmentation
• Complexity of Symbolic Representation in Working Memory of Transformer Correlates with the Complexity of a Task
• Around the GLOBE: Numerical Aggregation Question-Answering on Heterogeneous Genealogical Knowledge Graphs with Deep Neural Networks
• Can Transformers Reason About Effects of Actions?
• A Sliding-Window Approach to Automatic Creation of Meeting Minutes
• Multi-Scale 2D Temporal Adjacent Networks for Moment Localization with Natural Language
• Natural Language Inference in Context -- Investigating Contextual Reasoning over Long Texts
• VidTr: Video Transformer Without Convolutions
• Rissanen Data Analysis: Examining Dataset Characteristics via Description Length
• KVT: k-NN Attention for Boosting Vision Transformers
• Long-term series forecasting with Query Selector -- efficient model of sparse attention
• An Interpretable End-to-end Fine-tuning Approach for Long Clinical Text
• Deep Insights into Cognitive Decline: A Survey of Leveraging Non-Intrusive Modalities with Deep Learning Techniques
• Long-Range Transformer Architectures for Document Understanding
• Is In-hospital Meta-information Useful for Abstractive Discharge Summary Generation?
• Investigating the detection of Tortured Phrases in Scientific Literature
• Transformation Invariant Cancerous Tissue Classification Using Spatially Transformed DenseNet
• G-Transformer for Document-level Machine Translation
• Layered gradient accumulation and modular pipeline parallelism: fast and efficient training of large language models
• Staircase Attention for Recurrent Processing of Sequences
• LUKE-Graph: A Transformer-based Approach with Gated Relational Graph Attention for Cloze-style Reading Comprehension
• Multi-Task Prediction of Clinical Outcomes in the Intensive Care Unit using Flexible Multimodal Transformers
• Video Transformer for Deepfake Detection with Incremental Learning
• GLiT: Neural Architecture Search for Global and Local Image Transformer
• Improved Transformer for High-Resolution GANs
• Hi-Transformer: Hierarchical Interactive Transformer for Efficient and Effective Long Document Modeling
• Classifying Long Clinical Documents with Pre-trained Transformers
• ReadTwice: Reading Very Large Documents with Memories
• Optimizing small BERTs trained for German NER
• baller2vec++: A Look-Ahead Multi-Entity Transformer For Modeling Coordinated Agents
• MMCoVaR: Multimodal COVID-19 Vaccine Focused Data Repository for Fake News Detection and a Baseline Architecture for Classification
• Transfer Learning for Multi-lingual Tasks -- a Survey
• DebateSum: A large-scale argument mining and summarization dataset
• On Generating Extended Summaries of Long Documents
• PGT: Pseudo Relevance Feedback Using a Graph-Based Transformer
• Open Question Answering over Tables and Text
• Unsupervised Document Embedding via Contrastive Augmentation
• Conceptualizing Suicidal Behavior: Utilizing Explanations of Predicted Outcomes to Analyze Longitudinal Social Media Data
• LongKey: Keyphrase Extraction for Long Documents
• TANGNN: a Concise, Scalable and Effective Graph Neural Networks with Top-m Attention Mechanism for Graph Representation Learning
• Enhanced Labeling Technique for Reddit Text and Fine-Tuned Longformer Models for Classifying Depression Severity in English and Luganda
• BUSTER: a "BUSiness Transaction Entity Recognition" dataset
• N-ODE Transformer: A Depth-Adaptive Variant of the Transformer Using Neural Ordinary Differential Equations
• Long Document Ranking with Query-Directed Sparse Transformer
• DoSSIER@COLIEE 2021: Leveraging dense retrieval and summarization-based re-ranking for case law retrieval
• Pre-training Protein Language Models with Label-Agnostic Binding Pairs Enhances Performance in Downstream Tasks
• EasyTransfer -- A Simple and Scalable Deep Transfer Learning Platform for NLP Applications
• Global memory transformer for processing long documents
• Training Transformers for Information Security Tasks: A Case Study on Malicious URL Prediction
• Sparse Factorization of Large Square Matrices
• The Role of Global and Local Context in Named Entity Recognition
• EL-Attention: Memory Efficient Lossless Attention for Generation
• On Learning the Transformer Kernel
• Transformer-Based Behavioral Representation Learning Enables Transfer Learning for Mobile Sensing in Small Datasets
• On the Distribution, Sparsity, and Inference-time Quantization of Attention Values in Transformers
• Self-supervised Answer Retrieval on Clinical Notes
• Attention Mechanism with Energy-Friendly Operations
• TransMask: A Compact and Fast Speech Separation Model Based on Transformer
• Predicting COVID-19 Patient Shielding: A Comprehensive Study
• Shatter: An Efficient Transformer Encoder with Single-Headed Self-Attention and Relative Sequence Partitioning
• Dynamic Sliding Window for Meeting Summarization
• Smart Bird: Learnable Sparse Attention for Efficient and Effective Transformer
• Current Limitations of Language Models: What You Need is Retrieval
• DocNLI: A Large-scale Dataset for Document-level Natural Language Inference
• Adaptive Semiparametric Language Models
• SMYRF: Efficient Attention using Asymmetric Clustering
• Automating Chapter-Level Classification for Electronic Theses and Dissertations
• Multi-Scale Local-Temporal Similarity Fusion for Continuous Sign Language Recognition
• Token Pooling in Vision Transformers
• StreaMulT: Streaming Multimodal Transformer for Heterogeneous and Arbitrary Long Sequential Data
• TrackFormers: In Search of Transformer-Based Particle Tracking for the High-Luminosity LHC Era
• $\infty$-former: Infinite Memory Transformer
• Focus on the Core: Efficient Attention via Pruned Token Compression for Document Classification
• GroupLink: An End-to-end Multitask Method for Word Grouping and Relation Extraction in Form Understanding
• Can Deep Neural Networks Predict Data Correlations from Column Names?
• Beyond Nyströmformer -- Approximation of self-attention by Spectral Shifting
• Medical Code Prediction from Discharge Summary: Document to Sequence BERT using Sequence Attention
• Efficient Domain Adaptation of Language Models via Adaptive Tokenization
• Do Long-Range Language Models Actually Use Long-Range Context?
• Weakly-supervised Text Classification Based on Keyword Graph
• When Attention Meets Fast Recurrence: Training Language Models with Reduced Compute
• Multi-Domain Transformer-Based Counterfactual Augmentation for Earnings Call Analysis
• Natural Language Processing to Detect Cognitive Concerns in Electronic Health Records Using Deep Learning
• Linear Self-Attention Approximation via Trainable Feedforward Kernel
• Simple Hack for Transformers against Heavy Long-Text Classification on a Time- and Memory-Limited GPU Service
• Pre-trained Language Model based Ranking in Baidu Search
• The DEformer: An Order-Agnostic Distribution Estimating Transformer
• FastSeq: Make Sequence Generation Faster
• H-Transformer-1D: Fast One-Dimensional Hierarchical Attention for Sequences
• Interpretable Self-supervised Multi-task Learning for COVID-19 Information Retrieval and Extraction
• MORTY: Structured Summarization for Targeted Information Extraction from Scholarly Articles
• Description-based Label Attention Classifier for Explainable ICD-9 Classification
• Understanding and Overcoming the Challenges of Efficient Transformer Quantization
• Transfer training from smaller language model
• Input-independent Attention Weights Are Expressive Enough: A Study of Attention in Self-supervised Audio Transformers
• Efficient Transformer for Direct Speech Translation
• DCT: Dynamic Compressive Transformer for Modeling Unbounded Sequence
• HETFORMER: Heterogeneous Transformer with Sparse Attention for Long-Text Extractive Summarization
• Contrastive Document Representation Learning with Graph Attention Networks
• SciXGen: A Scientific Paper Dataset for Context-Aware Text Generation
• Skyformer: Remodel Self-Attention with Gaussian Kernel and Nyström Method
• BumbleBee: A Transformer for Music
• Domain Agnostic Few-Shot Learning For Document Intelligence
• ProSTformer: Pre-trained Progressive Space-Time Self-attention Model for Traffic Flow Forecasting
• Exploiting a Zoo of Checkpoints for Unseen Tasks
• DSBERT:Unsupervised Dialogue Structure learning with BERT
• SummerTime: Text Summarization Toolkit for Non-experts
• Data-Informed Global Sparseness in Attention Mechanisms for Deep Neural Networks
• Transformer Models for Text Coherence Assessment
• A Hierarchical Neural Framework for Classification and its Explanation in Large Unstructured Legal Documents
• PLSUM: Generating PT-BR Wikipedia by Summarizing Multiple Websites
• Decoupled Transformer for Scalable Inference in Open-domain Question Answering
• T-EMDE: Sketching-based global similarity for cross-modal retrieval
• FarFetched: Entity-centric Reasoning and Claim Validation for the Greek Language based on Textually Represented Environments
• What Makes a Star Teacher? A Hierarchical BERT Model for Evaluating Teacher's Performance in Online Education
• Searching Personal Collections
• Learning Hard Retrieval Decoder Attention for Transformers
• Knowledge Enhanced Sports Game Summarization
• Graph Convolutional Network for Swahili News Classification
• An Exploratory Study on Long Dialogue Summarization: What Works and What's Next
• DIALKI: Knowledge Identification in Conversational Systems through Dialogue-Document Contextualization
• Effective Distributed Representations for Academic Expert Search
• Corruption Is Not All Bad: Incorporating Discourse Structure into Pre-training via Corruption for Essay Scoring
• No Answer is Better Than Wrong Answer: A Reflection Model for Document Level Machine Reading Comprehension
• Learning Opinion Summarizers by Selecting Informative Reviews
• Vector-Vector-Matrix Architecture: A Novel Hardware-Aware Framework for Low-Latency Inference in NLP Applications
• On Inductive Biases for Machine Learning in Data Constrained Settings
• Corpus-level and Concept-based Explanations for Interpretable Document Classification
• Simulated Chats for Building Dialog Systems: Learning to Generate Conversations from Instructions
• Does Dialog Length matter for Next Response Selection task? An Empirical Study
• ReadOnce Transformers: Reusable Representations of Text for Transformers
• VAULT: VAriable Unified Long Text Representation for Machine Reading Comprehension
• PermuteFormer: Efficient Relative Position Encoding for Long Sequences
• A Neural Edge-Editing Approach for Document-Level Relation Graph Extraction
• Speech Summarization using Restricted Self-Attention
• Hierarchical Context-Aware Transformers for Non-Autoregressive Text to Speech
• Improving Patent Mining and Relevance Classification using Transformers
• Relation/Entity-Centric Reading Comprehension
• Can Transformer Models Measure Coherence In Text? Re-Thinking the Shuffle Test
• StreamHover: Livestream Transcript Summarization and Annotation
• RoR: Read-over-Read for Long Document Machine Reading Comprehension
• Highly Parallel Autoregressive Entity Linking with Discriminative Correction
• EfficientBERT: Progressively Searching Multilayer Perceptron via Warm-up Knowledge Distillation
• A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers
• Determining the Credibility of Science Communication
• Long-Range Modeling of Source Code Files with eWASH: Extended Window Access by Syntax Hierarchy
• Viola: A Topic Agnostic Generate-and-Rank Dialogue System
• MiRANews: Dataset and Benchmarks for Multi-Resource-Assisted News Summarization
• Revisiting Simple Neural Probabilistic Language Models
• Plot-guided Adversarial Example Construction for Evaluating Open-domain Story Generation
• Semantic Frame Forecast
• Don't be Contradicted with Anything! CI-ToD: Towards Benchmarking Consistency for Task-oriented Dialogue System
• Go Forth and Prosper: Language Modeling with Ancient Textual History