XLNet: Generalized Autoregressive Pretraining for Language Understanding

Abstract

With the capability of modeling bidirectional contexts, denoising autoencoding based pretraining like BERT achieves better performance than pretraining approaches based on autoregressive language modeling. However, relying on corrupting the input with masks, BERT neglects dependency between the masked positions and suffers from a pretrain-finetune discrepancy. In light of these pros and cons, we propose XLNet, a generalized autoregressive pretraining method that (1) enables learning bidirectional contexts by maximizing the expected likelihood over all permutations of the factorization order and (2) overcomes the limitations of BERT thanks to its autoregressive formulation. Furthermore, XLNet integrates ideas from Transformer-XL, the state-of-the-art autoregressive model, into pretraining. Empirically, under comparable experiment settings, XLNet outperforms BERT on 20 tasks, often by a large margin, including question answering, natural language inference, sentiment analysis, and document ranking.

Pretrained models and code are available at https://github.com/zihangdai/xlnet

References in corpus (21)

• Unsupervised Data Augmentation for Consistency Training
• Pixel Recurrent Neural Networks
• ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
• A Deep Relevance Matching Model for Ad-hoc Retrieval
• SQuAD: 100,000+ Questions for Machine Comprehension of Text
• Snorkel: Rapid Training Data Creation with Weak Supervision
• End-to-End Neural Ad-hoc Ranking with Kernel Pooling
• Semi-supervised Sequence Learning
• MaskGAN: Better Text Generation via Filling in the______
• MADE: Masked Autoencoder for Distribution Estimation
• Multi-Task Deep Neural Networks for Natural Language Understanding
• Adversarial Training Methods for Semi-Supervised Text Classification
• Word-Entity Duet Representations for Document Ranking
• RACE: Large-scale ReAding Comprehension Dataset From Examinations
• A Surprisingly Robust Trick for Winograd Schema Challenge
• NewsQA: A Machine Comprehension Dataset
• Breaking the Softmax Bottleneck: A High-Rank RNN Language Model
• Adaptive Input Representations for Neural Language Modeling
• Option Comparison Network for Multiple-choice Reading Comprehension
• Dual Co-Matching Network for Multi-choice Reading Comprehension
• Improving Question Answering with External Knowledge

Cited by in corpus (345)

• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Attention Mechanisms in Computer Vision: A Survey
• BERTScore: Evaluating Text Generation with BERT
• WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
• Pre-trained Models for Natural Language Processing: A Survey
• ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
• PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• ChatGPT and a New Academic Reality: Artificial Intelligence-Written Research Papers and the Ethics of the Large Language Models in Scholarly Publishing
• VL-BERT: Pre-training of Generic Visual-Linguistic Representations
• Multilingual Denoising Pre-training for Neural Machine Translation
• Q8BERT: Quantized 8Bit BERT
• Well-Read Students Learn Better: On the Importance of Pre-training Compact Models
• TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech
• Reducing Transformer Depth on Demand with Structured Dropout
• VLP: A Survey on Vision-Language Pre-training
• CodeBERT: A Pre-Trained Model for Programming and Natural Languages
• RoFormer: Enhanced Transformer with Rotary Position Embedding
• DocBERT: BERT for Document Classification
• Controllable Protein Design with Language Models
• ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation
• Portuguese Named Entity Recognition using BERT-CRF
• Incorporating BERT into Neural Machine Translation
• SMILES Transformer: Pre-trained Molecular Fingerprint for Low Data Drug Discovery
• Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey
• GraphCodeBERT: Pre-training Code Representations with Data Flow
• A Survey on Long-Tailed Visual Recognition
• ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Data
• Retrieving and Reading: A Comprehensive Survey on Open-domain Question Answering
• Deep Contextualized Acoustic Representations For Semi-Supervised Speech Recognition
• Regression Transformer: Concurrent sequence regression and generation for molecular language modeling
• Adapt or Get Left Behind: Domain Adaptation through BERT Language Model Finetuning for Aspect-Target Sentiment Classification
• ReClor: A Reading Comprehension Dataset Requiring Logical Reasoning
• Neural Networks for Entity Matching: A Survey
• Measuring and Reducing Gendered Correlations in Pre-trained Models
• MetaSleepLearner: A Pilot Study on Fast Adaptation of Bio-signals-Based Sleep Stage Classifier to New Individual Subject Using Meta-Learning
• Improving Transformer-based Speech Recognition Using Unsupervised Pre-training
• A Survey on Text Classification: From Shallow to Deep Learning
• Mixout: Effective Regularization to Finetune Large-scale Pretrained Language Models
• StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding
• Generating Accurate Assert Statements for Unit Test Cases using Pretrained Transformers
• Beyond 512 Tokens: Siamese Multi-depth Transformer-based Hierarchical Encoder for Long-Form Document Matching
• Pre-training via Paraphrasing
• Efficient Attention: Attention with Linear Complexities
• NEZHA: Neural Contextualized Representation for Chinese Language Understanding
• Low-Resource Knowledge-Grounded Dialogue Generation
• Language Models are Open Knowledge Graphs
• DepressionNet: A Novel Summarization Boosted Deep Framework for Depression Detection on Social Media
• Secure Evaluation of Quantized Neural Networks
• Overview of the TREC 2019 deep learning track
• Structured Pruning of a BERT-based Question Answering Model
• MisRoBÆRTa: Transformers versus Misinformation
• Rethinking Positional Encoding in Language Pre-training
• A Comparison of LSTM and BERT for Small Corpus
• Classification Aware Neural Topic Model and its Application on a New COVID-19 Disinformation Corpus
• MEANTIME: Mixture of Attention Mechanisms with Multi-temporal Embeddings for Sequential Recommendation
• Recent Advances in Natural Language Inference: A Survey of Benchmarks, Resources, and Approaches
• Med-BERT: pre-trained contextualized embeddings on large-scale structured electronic health records for disease prediction
• A Survey of Deep Learning for Scientific Discovery
• FastBERT: a Self-distilling BERT with Adaptive Inference Time
• Multi-scale Convolutional Neural Networks for Inverse Problems
• Gradient Boosting Neural Networks: GrowNet
• Generalization through Memorization: Nearest Neighbor Language Models
• MathBERT: A Pre-Trained Model for Mathematical Formula Understanding
• The Eighth Dialog System Technology Challenge
• Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT
• NAS-BERT: Task-Agnostic and Adaptive-Size BERT Compression with Neural Architecture Search
• Zero-Resource Knowledge-Grounded Dialogue Generation
• Using word embeddings to improve the discriminability of co-occurrence text networks
• CLUENER2020: Fine-grained Named Entity Recognition Dataset and Benchmark for Chinese
• Cross-lingual Retrieval for Iterative Self-Supervised Training
• Reweighted Proximal Pruning for Large-Scale Language Representation
• Explaining Question Answering Models through Text Generation
• Are Transformers universal approximators of sequence-to-sequence functions?
• Using Prior Knowledge to Guide BERT's Attention in Semantic Textual Matching Tasks
• Technical report on Conversational Question Answering
• Neural ranking models for document retrieval
• Recent progress in the JARVIS infrastructure for next-generation data-driven materials design
• FastMoE: A Fast Mixture-of-Expert Training System
• Automated Quality Assessment of Cognitive Behavioral Therapy Sessions Through Highly Contextualized Language Representations
• Factorized Multimodal Transformer for Multimodal Sequential Learning
• 12-in-1: Multi-Task Vision and Language Representation Learning
• Addressing "Documentation Debt" in Machine Learning Research: A Retrospective Datasheet for BookCorpus
• Pre-trained Language Models in Biomedical Domain: A Systematic Survey
• Are Pre-trained Language Models Aware of Phrases? Simple but Strong Baselines for Grammar Induction
• Robustness Verification for Transformers
• What Would Elsa Do? Freezing Layers During Transformer Fine-Tuning
• The Use of a Large Language Model for Cyberbullying Detection
• Dice Loss for Data-imbalanced NLP Tasks
• Corpus-Based Paraphrase Detection Experiments and Review
• IndoLEM and IndoBERT: A Benchmark Dataset and Pre-trained Language Model for Indonesian NLP
• LOREN: Logic-Regularized Reasoning for Interpretable Fact Verification
• Rethinking Positional Encoding
• Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping
• End-to-end Named Entity Recognition and Relation Extraction using Pre-trained Language Models
• Transformer-Based Models for Automatic Identification of Argument Relations: A Cross-Domain Evaluation
• SKEP: Sentiment Knowledge Enhanced Pre-training for Sentiment Analysis
• Outside the Box: Abstraction-Based Monitoring of Neural Networks
• Socially Enhanced Situation Awareness from Microblogs using Artificial Intelligence: A Survey
• TRANS-BLSTM: Transformer with Bidirectional LSTM for Language Understanding
• Tree-structured Attention with Hierarchical Accumulation
• PipeMare: Asynchronous Pipeline Parallel DNN Training
• Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language Model
• Fake or Genuine? Contextualised Text Representation for Fake Review Detection
• Learning to Encode Position for Transformer with Continuous Dynamical Model
• Ultra-High-Resolution Detector Simulation with Intra-Event Aware GAN and Self-Supervised Relational Reasoning
• Taking a Stance on Fake News: Towards Automatic Disinformation Assessment via Deep Bidirectional Transformer Language Models for Stance Detection
• A Review on the Applications of Transformer-based language models for Nucleotide Sequence Analysis
• Reducing Sentiment Bias in Language Models via Counterfactual Evaluation
• A Comparative Study of Transformer-Based Language Models on Extractive Question Answering
• The Effect of Natural Distribution Shift on Question Answering Models
• OPT: Omni-Perception Pre-Trainer for Cross-Modal Understanding and Generation
• Graph-Based Reasoning over Heterogeneous External Knowledge for Commonsense Question Answering
• EVA2.0: Investigating Open-Domain Chinese Dialogue Systems with Large-Scale Pre-Training
• CORD19STS: COVID-19 Semantic Textual Similarity Dataset
• XGPT: Cross-modal Generative Pre-Training for Image Captioning
• Semantic Networks for Engineering Design: A Survey
• TreeBERT: A Tree-Based Pre-Trained Model for Programming Language
• Certified Data Removal from Machine Learning Models
• DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference
• KELM: Knowledge Enhanced Pre-Trained Language Representations with Message Passing on Hierarchical Relational Graphs
• Pretrained AI Models: Performativity, Mobility, and Change
• SeMemNN: A Semantic Matrix-Based Memory Neural Network for Text Classification
• Automated Novelty Evaluation of Academic Paper: A Collaborative Approach Integrating Human and Large Language Model Knowledge
• Vision Transformers: From Semantic Segmentation to Dense Prediction
• Conformer-Kernel with Query Term Independence for Document Retrieval
• Evolution of transfer learning in natural language processing
• Speaker-Aware BERT for Multi-Turn Response Selection in Retrieval-Based Chatbots
• Fast Multi-Resolution Transformer Fine-tuning for Extreme Multi-label Text Classification
• On Linear Identifiability of Learned Representations
• Exploring Benefits of Transfer Learning in Neural Machine Translation
• ProTo: Program-Guided Transformer for Program-Guided Tasks
• Overview of the TREC 2022 deep learning track
• Diverse Image Inpainting with Bidirectional and Autoregressive Transformers
• Pre-Trained Models: Past, Present and Future
• Unsupervised Domain Adaptation of Language Models for Reading Comprehension
• Self-supervised Image-text Pre-training With Mixed Data In Chest X-rays
• DeepSeek-V3, GPT-4, Phi-4, and LLaMA-3.3 generate correct code for LoRaWAN-related engineering tasks
• SOLD: Sinhala Offensive Language Dataset
• Towards Domain Adaptation from Limited Data for Question Answering Using Deep Neural Networks
• EmotionIC: emotional inertia and contagion-driven dependency modeling for emotion recognition in conversation
• Look at What I'm Doing: Self-Supervised Spatial Grounding of Narrations in Instructional Videos
• Pre-Trained and Attention-Based Neural Networks for Building Noetic Task-Oriented Dialogue Systems
• QUEACO: Borrowing Treasures from Weakly-labeled Behavior Data for Query Attribute Value Extraction
• "When they say weed causes depression, but it's your fav antidepressant": Knowledge-aware Attention Framework for Relationship Extraction
• Taming Pretrained Transformers for Extreme Multi-label Text Classification
• Cracking the Contextual Commonsense Code: Understanding Commonsense Reasoning Aptitude of Deep Contextual Representations
• LMSanitator: Defending Prompt-Tuning Against Task-Agnostic Backdoors
• Neuro-symbolic Architectures for Context Understanding
• DeepSI: Interactive Deep Learning for Semantic Interaction
• Improved Hierarchical Patient Classification with Language Model Pretraining over Clinical Notes
• Transfer Learning from Transformers to Fake News Challenge Stance Detection (FNC-1) Task
• Analyzing the Forgetting Problem in the Pretrain-Finetuning of Dialogue Response Models
• MC-BERT: Efficient Language Pre-Training via a Meta Controller
• Spatial-Temporal Self-Attention Network for Flow Prediction
• On the comparability of Pre-trained Language Models
• NLP From Scratch Without Large-Scale Pretraining: A Simple and Efficient Framework
• Improve Transformer Models with Better Relative Position Embeddings
• $O(n)$ Connections are Expressive Enough: Universal Approximability of Sparse Transformers
• Learning Norms from Stories: A Prior for Value Aligned Agents
• Enabling Language Models to Fill in the Blanks
• Training Multilingual Pre-trained Language Model with Byte-level Subwords
• Defending Against Backdoor Attacks in Natural Language Generation
• Lacking the embedding of a word? Look it up into a traditional dictionary
• Real-Time Execution of Large-scale Language Models on Mobile
• SCARF: Self-Supervised Contrastive Learning using Random Feature Corruption
• Multi-task Batch Reinforcement Learning with Metric Learning
• Retrieval-Augmented Transformer-XL for Close-Domain Dialog Generation
• Unsupervised Editing for Counterfactual Stories
• Mining Implicit Relevance Feedback from User Behavior for Web Question Answering
• VIOLIN: A Large-Scale Dataset for Video-and-Language Inference
• DocVQA: A Dataset for VQA on Document Images
• Scatterbrain: Unifying Sparse and Low-rank Attention Approximation
• Literature review on vulnerability detection using NLP technology
• DeepTrax: Embedding Graphs of Financial Transactions
• LightPAFF: A Two-Stage Distillation Framework for Pre-training and Fine-tuning
• Variance-reduced Language Pretraining via a Mask Proposal Network
• Multi-Stage Conversational Passage Retrieval: An Approach to Fusing Term Importance Estimation and Neural Query Rewriting
• Denoised Internal Models: a Brain-Inspired Autoencoder against Adversarial Attacks
• Adversarial NLI for Factual Correctness in Text Summarisation Models
• Disentangling Adaptive Gradient Methods from Learning Rates
• Fusing Context Into Knowledge Graph for Commonsense Question Answering
• Sketch-BERT: Learning Sketch Bidirectional Encoder Representation from Transformers by Self-supervised Learning of Sketch Gestalt
• Sentence Analogies: Exploring Linguistic Relationships and Regularities in Sentence Embeddings
• Entity and Evidence Guided Relation Extraction for DocRED
• Vokenization: Improving Language Understanding with Contextualized, Visual-Grounded Supervision
• Multi-node Bert-pretraining: Cost-efficient Approach
• Evaluation of Sentence Representations in Polish
• A Comparison of Pre-trained Vision-and-Language Models for Multimodal Representation Learning across Medical Images and Reports
• CoTexT: Multi-task Learning with Code-Text Transformer
• Efficient Softmax Approximation for Deep Neural Networks with Attention Mechanism
• Hide-and-Seek: A Template for Explainable AI
• E-Stitchup: Data Augmentation for Pre-Trained Embeddings
• Detecting and analyzing missing citations to published scientific entities
• Question Generation by Transformers
• On the Impact of Knowledge-based Linguistic Annotations in the Quality of Scientific Embeddings
• Symmetric Regularization based BERT for Pair-wise Semantic Reasoning
• An Analysis on Matching Mechanisms and Token Pruning for Late-interaction Models
• Beyond English-Only Reading Comprehension: Experiments in Zero-Shot Multilingual Transfer for Bulgarian
• InstructNet: A Novel Approach for Multi-Label Instruction Classification through Advanced Deep Learning
• Robustly Pre-trained Neural Model for Direct Temporal Relation Extraction
• DLGNet: A Transformer-based Model for Dialogue Response Generation
• Applying Transfer Learning for Improving Domain-Specific Search Experience Using Query to Question Similarity
• Generative Deep Learning Techniques for Password Generation
• LRC-BERT: Latent-representation Contrastive Knowledge Distillation for Natural Language Understanding
• Evolving Character-level Convolutional Neural Networks for Text Classification
• TTTTTackling WinoGrande Schemas
• FMMformer: Efficient and Flexible Transformer via Decomposed Near-field and Far-field Attention
• How Can BERT Help Lexical Semantics Tasks?
• The CoRa Tensor Compiler: Compilation for Ragged Tensors with Minimal Padding
• An Algorithm for Routing Capsules in All Domains
• Efficient Nearest Neighbor Language Models
• Using Transformers to Provide Teachers with Personalized Feedback on their Classroom Discourse: The TalkMoves Application
• NUIG-Shubhanker@Dravidian-CodeMix-FIRE2020: Sentiment Analysis of Code-Mixed Dravidian text using XLNet
• Cluster-based Deep Ensemble Learning for Emotion Classification in Internet Memes
• Curb Your Carbon Emissions: Benchmarking Carbon Emissions in Machine Translation
• memeBot: Towards Automatic Image Meme Generation
• Low-Shot Classification: A Comparison of Classical and Deep Transfer Machine Learning Approaches
• Syntax-driven Iterative Expansion Language Models for Controllable Text Generation
• PatentTransformer-2: Controlling Patent Text Generation by Structural Metadata
• Exploring Transformers in Emotion Recognition: a comparison of BERT, DistillBERT, RoBERTa, XLNet and ELECTRA
• Effective Transfer Learning for Identifying Similar Questions: Matching User Questions to COVID-19 FAQs
• Optimizing Deeper Transformers on Small Datasets
• Audio MFCC-gram Transformers for respiratory insufficiency detection in COVID-19
• Position Masking for Language Models
• Natural Language Inference in Context -- Investigating Contextual Reasoning over Long Texts
• A Comparative Study of Lexical Substitution Approaches based on Neural Language Models
• Explicit Pairwise Word Interaction Modeling Improves Pretrained Transformers for English Semantic Similarity Tasks
• MVP-BERT: Redesigning Vocabularies for Chinese BERT and Multi-Vocab Pretraining
• Heads-up! Unsupervised Constituency Parsing via Self-Attention Heads
• A Survey of Quantum Theory Inspired Approaches to Information Retrieval
• Conversational Question Answering: A Survey
• Invenio: Discovering Hidden Relationships Between Tasks/Domains Using Structured Meta Learning
• Neural, Symbolic and Neural-Symbolic Reasoning on Knowledge Graphs
• How to Select One Among All? An Extensive Empirical Study Towards the Robustness of Knowledge Distillation in Natural Language Understanding
• Complex Transformer: A Framework for Modeling Complex-Valued Sequence
• Unsupervised Pre-training for Natural Language Generation: A Literature Review
• DEUX: An Attribute-Guided Framework for Sociable Recommendation Dialog Systems
• Optimizing small BERTs trained for German NER
• baller2vec++: A Look-Ahead Multi-Entity Transformer For Modeling Coordinated Agents
• Transfer Learning for Multi-lingual Tasks -- a Survey
• Data-driven models and computational tools for neurolinguistics: a language technology perspective
• CMV-BERT: Contrastive multi-vocab pretraining of BERT
• Transfer Learning Robustness in Multi-Class Categorization by Fine-Tuning Pre-Trained Contextualized Language Models
• Does BERT Understand Sentiment? Leveraging Comparisons Between Contextual and Non-Contextual Embeddings to Improve Aspect-Based Sentiment Models
• Neural Composition: Learning to Generate from Multiple Models
• Understanding Semantics from Speech Through Pre-training
• Improving Diversity of Neural Text Generation via Inverse Probability Weighting
• A Pairwise Probe for Understanding BERT Fine-Tuning on Machine Reading Comprehension
• Current Limitations of Language Models: What You Need is Retrieval
• Billion-scale Pre-trained E-commerce Product Knowledge Graph Model
• An Empirical Investigation of Contextualized Number Prediction
• A Qualitative Evaluation of Language Models on Automatic Question-Answering for COVID-19
• Contextualized End-to-End Neural Entity Linking
• An Experimental Evaluation of Transformer-based Language Models in the Biomedical Domain
• Bilingual Language Modeling, A transfer learning technique for Roman Urdu
• A Mixture of $h-1$ Heads is Better than $h$ Heads
• JASS: Japanese-specific Sequence to Sequence Pre-training for Neural Machine Translation
• Spoiler Alert: Using Natural Language Processing to Detect Spoilers in Book Reviews
• Masked ELMo: An evolution of ELMo towards fully contextual RNN language models
• Can You Fool AI by Doing a 180? $\unicode{x2013}$ A Case Study on Authorship Analysis of Texts by Arata Osada
• An Effective Contextual Language Modeling Framework for Speech Summarization with Augmented Features
• Alignment Attention by Matching Key and Query Distributions
• Quantifying the Task-Specific Information in Text-Based Classifications
• Exploring Fluent Query Reformulations with Text-to-Text Transformers and Reinforcement Learning
• Cocktail: Leveraging Ensemble Learning for Optimized Model Serving in Public Cloud
• On the Importance of Adaptive Data Collection for Extremely Imbalanced Pairwise Tasks
• Integrating Lexical Knowledge in Word Embeddings using Sprinkling and Retrofitting
• Modeling Inter-Speaker Relationship in XLNet for Contextual Spoken Language Understanding
• Fine-tuning Pre-trained Contextual Embeddings for Citation Content Analysis in Scholarly Publication
• A Deep Recurrent Survival Model for Unbiased Ranking
• Enhancing Review Comprehension with Domain-Specific Commonsense
• A Short Survey of Pre-trained Language Models for Conversational AI-A NewAge in NLP
• Larger-Context Tagging: When and Why Does It Work?
• Retraining DistilBERT for a Voice Shopping Assistant by Using Universal Dependencies
• Quantum-inspired Multimodal Fusion for Video Sentiment Analysis
• Improved Customer Transaction Classification using Semi-Supervised Knowledge Distillation
• KoreALBERT: Pretraining a Lite BERT Model for Korean Language Understanding
• Learning to Emphasize: Dataset and Shared Task Models for Selecting Emphasis in Presentation Slides
• ActBERT: Learning Global-Local Video-Text Representations
• Multiplicative Position-aware Transformer Models for Language Understanding
• Pre-training for Ad-hoc Retrieval: Hyperlink is Also You Need
• PairConnect: A Compute-Efficient MLP Alternative to Attention
• TVDIM: Enhancing Image Self-Supervised Pretraining via Noisy Text Data
• Multilingual Medical Question Answering and Information Retrieval for Rural Health Intelligence Access
• Improving the quality of Persian clinical text with a novel spelling correction system
• Applying Text Mining to Analyze Human Question Asking in Creativity Research
• An Overview on Generative AI at Scale with Edge-Cloud Computing
• Should We Be Pre-training? An Argument for End-task Aware Training as an Alternative
• R&R: Metric-guided Adversarial Sentence Generation
• Mischief: A Simple Black-Box Attack Against Transformer Architectures
• Finnish Language Modeling with Deep Transformer Models
• WIKIR: A Python toolkit for building a large-scale Wikipedia-based English Information Retrieval Dataset
• Pre-Trained Models for Heterogeneous Information Networks
• Pre-trained Language Model Based Active Learning for Sentence Matching
• Bi-directional Cognitive Thinking Network for Machine Reading Comprehension
• Modelling General Properties of Nouns by Selectively Averaging Contextualised Embeddings
• When to Fold'em: How to answer Unanswerable questions
• Cross-language Information Retrieval
• An Empirical Study of Topic Transition in Dialogue
• Using Artificial Intelligence to Augment Science Prioritization for Astro2020
• Integrating Knowledge into End-to-End Speech Recognition from External Text-Only Data
• Answer Generation for Questions With Multiple Information Sources in E-Commerce
• Action based Network for Conversation Question Reformulation
• Towards a Transformer-Based Reverse Dictionary Model for Quality Estimation of Definitions
• Normalization of Input-output Shared Embeddings in Text Generation Models
• Occlusion Aware Kernel Correlation Filter Tracker using RGB-D
• Leveraging Linguistic Coordination in Reranking N-Best Candidates For End-to-End Response Selection Using BERT
• Why Can You Lay Off Heads? Investigating How BERT Heads Transfer
• LV-BERT: Exploiting Layer Variety for BERT
• The Brownian motion in the transformer model
• Knowledge Transfer by Discriminative Pre-training for Academic Performance Prediction
• Musical Speech: A Transformer-based Composition Tool
• Semantic Answer Type Prediction using BERT: IAI at the ISWC SMART Task 2020
• Unsupervised Keyphrase Extraction by Jointly Modeling Local and Global Context
• Don't be Contradicted with Anything! CI-ToD: Towards Benchmarking Consistency for Task-oriented Dialogue System
• Small-Bench NLP: Benchmark for small single GPU trained models in Natural Language Processing
• One-shot Key Information Extraction from Document with Deep Partial Graph Matching
• What makes us curious? analysis of a corpus of open-domain questions
• Towards Preference Learning for Autonomous Ground Robot Navigation Tasks
• Character-level Representations Improve DRS-based Semantic Parsing Even in the Age of BERT
• IIT_kgp at FinCausal 2020, Shared Task 1: Causality Detection using Sentence Embeddings in Financial Reports
• Multiple Structural Priors Guided Self Attention Network for Language Understanding
• Improving speech recognition models with small samples for air traffic control systems
• Few Shot Learning for Information Verification
• LRG at TREC 2020: Document Ranking with XLNet-Based Models
• Learning a Word-Level Language Model with Sentence-Level Noise Contrastive Estimation for Contextual Sentence Probability Estimation
• Evaluating Document Coherence Modelling
• BERT Embeddings Can Track Context in Conversational Search
• DFraud3- Multi-Component Fraud Detection freeof Cold-start
• Emotion Correlation Mining Through Deep Learning Models on Natural Language Text
• Using the Hammer Only on Nails: A Hybrid Method for Evidence Retrieval for Question Answering
• No Answer is Better Than Wrong Answer: A Reflection Model for Document Level Machine Reading Comprehension
• MaP: A Matrix-based Prediction Approach to Improve Span Extraction in Machine Reading Comprehension
• Examining the rhetorical capacities of neural language models
• Effective Unsupervised Domain Adaptation with Adversarially Trained Language Models
• A Morpho-Syntactically Informed LSTM-CRF Model for Named Entity Recognition
• Pre-train, Interact, Fine-tune: A Novel Interaction Representation for Text Classification
• BIG MOOD: Relating Transformers to Explicit Commonsense Knowledge
• Towards Learning Cross-Modal Perception-Trace Models
• A neural document language modeling framework for spoken document retrieval
• Multi-label Classification for Automatic Tag Prediction in the Context of Programming Challenges
• Memeify: A Large-Scale Meme Generation System
• Transformer-based language modeling and decoding for conversational speech recognition
• Fine-tuning Multi-hop Question Answering with Hierarchical Graph Network