Recent Trends in Deep Learning Based Natural Language Processing

Abstract

Deep learning methods employ multiple processing layers to learn hierarchical representations of data and have produced state-of-the-art results in many domains. Recently, a variety of model designs and methods have blossomed in the context of natural language processing (NLP). In this paper, we review significant deep learning related models and methods that have been employed for numerous NLP tasks and provide a walk-through of their evolution. We also summarize, compare and contrast the various models and put forward a detailed understanding of the past, present and future of deep learning in NLP.

arXiv commentAdded BERT, ELMo, Transformer

References in corpus (48)

• Efficient Estimation of Word Representations in Vector Space
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Natural Language Processing (almost) from Scratch
• Caffe: Convolutional Architecture for Fast Feature Embedding
• From Frequency to Meaning: Vector Space Models of Semantics
• Convolutional Sequence to Sequence Learning
• Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
• A Neural Conversational Model
• A Deep Reinforced Model for Abstractive Summarization
• Enhanced LSTM for Natural Language Inference
• Language Modeling with Gated Convolutional Networks
• Character-Aware Neural Language Models
• Sequence Level Training with Recurrent Neural Networks
• Comparative Study of CNN and RNN for Natural Language Processing
• Long Short-Term Memory Based Recurrent Neural Network Architectures for Large Vocabulary Speech Recognition
• SQuAD: 100,000+ Questions for Machine Comprehension of Text
• Dynamic Memory Networks for Visual and Textual Question Answering
• CNN Features off-the-shelf: an Astounding Baseline for Recognition
• word2vec Parameter Learning Explained
• Ask Me Anything: Dynamic Memory Networks for Natural Language Processing
• Transition-Based Dependency Parsing with Stack Long Short-Term Memory
• A large annotated corpus for learning natural language inference
• Enriching Word Vectors with Subword Information
• QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension
• Deep Reinforcement Learning for Dialogue Generation
• A Deeper Look into Sarcastic Tweets Using Deep Convolutional Neural Networks
• ReasoNet: Learning to Stop Reading in Machine Comprehension
• Adversarial Learning for Neural Dialogue Generation
• Ask Your Neurons: A Neural-based Approach to Answering Questions about Images
• A Persona-Based Neural Conversation Model
• Multimodal Machine Learning: A Survey and Taxonomy
• Toward Controlled Generation of Text
• Adversarial Generation of Natural Language
• Modelling, Visualising and Summarising Documents with a Single Convolutional Neural Network
• Reading Wikipedia to Answer Open-Domain Questions
• Open Question Answering with Weakly Supervised Embedding Models
• On-line Active Reward Learning for Policy Optimisation in Spoken Dialogue Systems
• Aspect Level Sentiment Classification with Deep Memory Network
• A Neural Network Approach to Context-Sensitive Generation of Conversational Responses
• Evaluating Prerequisite Qualities for Learning End-to-End Dialog Systems
• Learning from Real Users: Rating Dialogue Success with Neural Networks for Reinforcement Learning in Spoken Dialogue Systems
• Modeling Relational Information in Question-Answer Pairs with Convolutional Neural Networks
• Structured Training for Neural Network Transition-Based Parsing
• Stochastic Answer Networks for Machine Reading Comprehension
• Are distributional representations ready for the real world? Evaluating word vectors for grounded perceptual meaning
• Beyond Bilingual: Multi-sense Word Embeddings using Multilingual Context
• Context-Dependent Translation Selection Using Convolutional Neural Network

Cited by in corpus (193)

• Deep Learning Algorithms for Rotating Machinery Intelligent Diagnosis: An Open Source Benchmark Study
• Attention in Natural Language Processing
• Large-Scale Optical Neural Networks based on Photoelectric Multiplication
• Graph-based Deep Learning for Communication Networks: A Survey
• Pruning by Explaining: A Novel Criterion for Deep Neural Network Pruning
• explAIner: A Visual Analytics Framework for Interactive and Explainable Machine Learning
• Adversarial Attacks on Deep Learning Models in Natural Language Processing: A Survey
• Medical Deep Learning -- A systematic Meta-Review
• Deep Learning for Generic Object Detection: A Survey
• Unsupervised Scalable Representation Learning for Multivariate Time Series
• NL4DV: A Toolkit for Generating Analytic Specifications for Data Visualization from Natural Language Queries
• DocBERT: BERT for Document Classification
• Controllable Protein Design with Language Models
• A Survey on Methods and Theories of Quantized Neural Networks
• Natural Language Processing Advancements By Deep Learning: A Survey
• Deep learning at the shallow end: Malware classification for non-domain experts
• Towards Natural Language Interfaces for Data Visualization: A Survey
• Patient2Vec: A Personalized Interpretable Deep Representation of the Longitudinal Electronic Health Record
• Ablation Studies in Artificial Neural Networks
• FlowSense: A Natural Language Interface for Visual Data Exploration within a Dataflow System
• Transfer Learning in Deep Reinforcement Learning: A Survey
• PFNN: A Penalty-Free Neural Network Method for Solving a Class of Second-Order Boundary-Value Problems on Complex Geometries
• Neural Entity Linking: A Survey of Models Based on Deep Learning
• The Modern Mathematics of Deep Learning
• Recurrent Neural Networks: An Embedded Computing Perspective
• Suicidal Ideation and Mental Disorder Detection with Attentive Relation Networks
• A Deep Neural Network for SSVEP-based Brain-Computer Interfaces
• Continual Learning for Recurrent Neural Networks: an Empirical Evaluation
• Real-Time Detection of Gravitational Waves from Binary Neutron Stars using Artificial Neural Networks
• Recent Advances in Deep Learning: An Overview
• Improved Automatic Summarization of Subroutines via Attention to File Context
• A Survey of Deep Learning Techniques for Neural Machine Translation
• Achieving Super-Linear Speedup across Multi-FPGA for Real-Time DNN Inference
• Natural quantum reservoir computing for temporal information processing
• Why gradient clipping accelerates training: A theoretical justification for adaptivity
• Julia Language in Machine Learning: Algorithms, Applications, and Open Issues
• Local Aggregation for Unsupervised Learning of Visual Embeddings
• Digital Twin: Values, Challenges and Enablers
• Information Processing Capacity of Spin-Based Quantum Reservoir Computing Systems
• Data driven approximation of parametrized PDEs by Reduced Basis and Neural Networks
• Do Explanations Reflect Decisions? A Machine-centric Strategy to Quantify the Performance of Explainability Algorithms
• Unleashing the power of disruptive and emerging technologies amid COVID-19: A detailed review
• Detection and Parameter Estimation of Gravitational Waves from Binary Neutron-Star Mergers in Real LIGO Data using Deep Learning
• Ensembles of Convolutional Neural Networks models for pediatric pneumonia diagnosis
• Deep Feature Space: A Geometrical Perspective
• Data-driven control of room temperature and bidirectional EV charging using deep reinforcement learning: simulations and experiments
• Learning to Continuously Optimize Wireless Resource in a Dynamic Environment: A Bilevel Optimization Perspective
• Scalable Deep Learning on Distributed Infrastructures: Challenges, Techniques and Tools
• Natural Language Interfaces to Data
• CodeGRU: Context-aware Deep Learning with Gated Recurrent Unit for Source Code Modeling
• Spatio-Temporal Graph Transformer Networks for Pedestrian Trajectory Prediction
• Passport-aware Normalization for Deep Model Protection
• A Comprehensive Survey of Machine Learning Applied to Radar Signal Processing
• Unsupervised Representation Learning for Time Series with Temporal Neighborhood Coding
• Deep Ensemble Analysis for Imaging X-ray Polarimetry
• Why Do Deep Residual Networks Generalize Better than Deep Feedforward Networks? -- A Neural Tangent Kernel Perspective
• ExPAN(N)D: Exploring Posits for Efficient Artificial Neural Network Design in FPGA-based Systems
• Early Disease Stage Characterization in Parkinson's Disease from Resting-state fMRI Data Using a Long Short-term Memory Network
• Nemesyst: A Hybrid Parallelism Deep Learning-Based Framework Applied for Internet of Things Enabled Food Retailing Refrigeration Systems
• Protocol Discovery for the Quantum Control of Majoranas by Differentiable Programming and Natural Evolution Strategies
• Fake or Genuine? Contextualised Text Representation for Fake Review Detection
• Detecting Requirements Smells With Deep Learning: Experiences, Challenges and Future Work
• Embedding Human Heuristics in Machine-Learning-Enabled Probe Microscopy
• Opening the black box of deep learning
• ConFoc: Content-Focus Protection Against Trojan Attacks on Neural Networks
• Learning to Continuously Optimize Wireless Resource In Episodically Dynamic Environment
• Creating user stereotypes for persona development from qualitative data through semi-automatic subspace clustering
• FarsTail: A Persian Natural Language Inference Dataset
• Deep Neural Network Fingerprinting by Conferrable Adversarial Examples
• AngularGrad: A New Optimization Technique for Angular Convergence of Convolutional Neural Networks
• Visualizing Graph Neural Networks with CorGIE: Corresponding a Graph to Its Embedding
• Towards non-toxic landscapes: Automatic toxic comment detection using DNN
• Covid-Transformer: Detecting COVID-19 Trending Topics on Twitter Using Universal Sentence Encoder
• A Dimensionality Reduction Approach for Convolutional Neural Networks
• Conditional Generative Neural System for Probabilistic Trajectory Prediction
• Australia's long-term electricity demand forecasting using deep neural networks
• Deep Learning in Protein Structural Modeling and Design
• A Scalable Framework for Multilevel Streaming Data Analytics using Deep Learning
• Open Sesame: Getting Inside BERT's Linguistic Knowledge
• Deep learning for photoacoustic imaging: a survey
• A Novel Framework Integrating AI Model and Enzymological Experiments Promotes Identification of SARS-CoV-2 3CL Protease Inhibitors and Activity-based Probe
• Vau da muntanialas: Energy-efficient multi-die scalable acceleration of RNN inference
• Machine Learning for Detecting Data Exfiltration: A Review
• Disentangled Variational Auto-Encoder for Semi-supervised Learning
• Plasma Image Classification Using Cosine Similarity Constrained CNN
• MLGOPerf: An ML Guided Inliner to Optimize Performance
• Sequential Interpretability: Methods, Applications, and Future Direction for Understanding Deep Learning Models in the Context of Sequential Data
• 3D Convolutional Neural Networks for Ultrasound-Based Silent Speech Interfaces
• A Context-based Disambiguation Model for Sentiment Concepts Using a Bag-of-concepts Approach
• Convolutional Neural Network based Multiple-Rate Compressive Sensing for Massive MIMO CSI Feedback: Design, Simulation, and Analysis
• Analyzing the Forgetting Problem in the Pretrain-Finetuning of Dialogue Response Models
• Inference of neutrino flavor evolution through data assimilation and neural differential equations
• A Sufficient Statistic for Influence in Structured Multiagent Environments
• DBTagger: Multi-Task Learning for Keyword Mapping in NLIDBs Using Bi-Directional Recurrent Neural Networks
• Neural model robustness for skill routing in large-scale conversational AI systems: A design choice exploration
• Neural Baselines for Word Alignment
• Evaluating KGR10 Polish word embeddings in the recognition of temporal expressions using BiLSTM-CRF
• Combining Pre-trained Word Embeddings and Linguistic Features for Sequential Metaphor Identification
• A Survey on Machine Reading Comprehension: Tasks, Evaluation Metrics and Benchmark Datasets
• Speech Pattern based Black-box Model Watermarking for Automatic Speech Recognition
• MetaPred: Meta-Learning for Clinical Risk Prediction with Limited Patient Electronic Health Records
• Attention-based Neural Load Forecasting: A Dynamic Feature Selection Approach
• A Unified Transformer-based Framework for Duplex Text Normalization
• Local Label Propagation for Large-Scale Semi-Supervised Learning
• Reconfigurable Intelligent Surface Enabled Federated Learning: A Unified Communication-Learning Design Approach
• ProtoryNet - Interpretable Text Classification Via Prototype Trajectories
• Adversarial Attacks and Defenses: An Interpretation Perspective
• Utilising Graph Machine Learning within Drug Discovery and Development
• Sanity Check: A Strong Alignment and Information Retrieval Baseline for Question Answering
• Text Analysis in Adversarial Settings: Does Deception Leave a Stylistic Trace?
• A Novel Fusion of Attention and Sequence to Sequence Autoencoders to Predict Sleepiness From Speech
• Classification of Brainwave Signals Based on Hybrid Deep Learning and an Evolutionary Algorithm
• Task Scheduling for Simultaneous IoT Sensing and Energy Harvesting: A Survey and Critical Analysis
• An Effective Label Noise Model for DNN Text Classification
• Patent Analytics Based on Feature Vector Space Model: A Case of IoT
• A Pre-defined Sparse Kernel Based Convolution for Deep CNNs
• High-dimensional approximation spaces of artificial neural networks and applications to partial differential equations
• Multimodal Classification of Teaching Activities from University Lecture Recordings
• A Scalable Multilabel Classification to Deploy Deep Learning Architectures For Edge Devices
• On the Distributional Properties of Adaptive Gradients
• Exploring Structure Consistency for Deep Model Watermarking
• Learning from Learning Machines: Optimisation, Rules, and Social Norms
• Cellular Traffic Prediction with Recurrent Neural Network
• Multi Task Deep Morphological Analyzer: Context Aware Joint Morphological Tagging and Lemma Prediction
• Classifying CMB time-ordered data through deep neural networks
• Automated Query Reformulation for Efficient Search based on Query Logs From Stack Overflow
• Combining Spiking Neural Network and Artificial Neural Network for Enhanced Image Classification
• A survey on extremism analysis using Natural Language Processing
• Enhancing Sindhi Word Segmentation using Subword Representation Learning and Position-aware Self-attention
• The Implicit Bias for Adaptive Optimization Algorithms on Homogeneous Neural Networks
• Intrinsic Geometric Vulnerability of High-Dimensional Artificial Intelligence
• A little goes a long way: Improving toxic language classification despite data scarcity
• New Ideas and Trends in Deep Multimodal Content Understanding: A Review
• On the impressive performance of randomly weighted encoders in summarization tasks
• Encoding-based Memory Modules for Recurrent Neural Networks
• Neural network for multi-exponential sound energy decay analysis
• Word Embedding Algorithms as Generalized Low Rank Models and their Canonical Form
• Automated Source Code Generation and Auto-completion Using Deep Learning: Comparing and Discussing Current Language-Model-Related Approaches
• Machine Learning Based on Natural Language Processing to Detect Cardiac Failure in Clinical Narratives
• Weighted graphlets and deep neural networks for protein structure classification
• Multimodal Sentiment Analysis with Word-Level Fusion and Reinforcement Learning
• Active Subspace of Neural Networks: Structural Analysis and Universal Attacks
• Using Artificial Intelligence to Recapture Norms: Did #metoo change gender norms in Sweden?
• Short-term probabilistic photovoltaic power forecast based on deep convolutional long short-term memory network and kernel density estimation
• Enhanced Memory Network: The novel network structure for Symbolic Music Generation
• Ranking and Rejecting of Pre-Trained Deep Neural Networks in Transfer Learning based on Separation Index
• Robustness of on-device Models: Adversarial Attack to Deep Learning Models on Android Apps
• BERT-based knowledge extraction method of unstructured domain text
• Intelligent requirements engineering from natural language and their chaining toward CAD models
• A Visual Analytics Framework for Explaining and Diagnosing Transfer Learning Processes
• Deconstructing Generative Adversarial Networks
• Prosody Learning Mechanism for Speech Synthesis System Without Text Length Limit
• Adversarial Learning for Supervised and Semi-supervised Relation Extraction in Biomedical Literature
• Joint Design of Radar Waveform and Detector via End-to-end Learning with Waveform Constraints
• Towards Recognizing New Semantic Concepts in New Visual Domains
• aiSTROM -- A roadmap for developing a successful AI strategy
• BEAN: Interpretable Representation Learning with Biologically-Enhanced Artificial Neuronal Assembly Regularization
• Imitating Deep Learning Dynamics via Locally Elastic Stochastic Differential Equations
• Analyzing Structures in the Semantic Vector Space: A Framework for Decomposing Word Embeddings
• Dataset Bias Mitigation Through Analysis of CNN Training Scores
• SQWA: Stochastic Quantized Weight Averaging for Improving the Generalization Capability of Low-Precision Deep Neural Networks
• Pretrained Language Models for Dialogue Generation with Multiple Input Sources
• Immuno-mimetic Deep Neural Networks (Immuno-Net)
• Comparing Acoustic-based Approaches for Alzheimer's Disease Detection
• Deep Learning -- A first Meta-Survey of selected Reviews across Scientific Disciplines, their Commonalities, Challenges and Research Impact
• Sample Prior Guided Robust Model Learning to Suppress Noisy Labels
• Design Considerations For Hypothesis Rejection Modules In Spoken Language Understanding Systems
• Explaining Adversarial Vulnerability with a Data Sparsity Hypothesis
• Exploring difference in public perceptions on HPV vaccine between gender groups from Twitter using deep learning
• Preprint: Norm Loss: An efficient yet effective regularization method for deep neural networks
• Recurrent convolutional neural networks for mandible segmentation from computed tomography
• Edge service resource allocation strategy based on intelligent prediction
• Approximate Random Dropout
• Machine learning-based porosity estimation from spectral decomposed seismic data
• Analyzing users' sentiment towards popular consumer industries and brands on Twitter
• A Recurrent Neural Network Based Patch Recommender for Linux Kernel Bugs
• TATi-Thermodynamic Analytics ToolkIt: TensorFlow-based software for posterior sampling in machine learning applications
• From web crawled text to project descriptions: automatic summarizing of social innovation projects
• Deep Learning Based Natural Language Processing for End to End Speech Translation
• Learning Graph Neural Networks with Approximate Gradient Descent
• TEDL: A Text Encryption Method Based on Deep Learning
• Invisible Marker: Automatic Annotation of Segmentation Masks for Object Manipulation
• Deep Reinforcement Learning for Dynamic Spectrum Sensing and Aggregation in Multi-Channel Wireless Networks
• Visual Summary of Value-level Feature Attribution in Prediction Classes with Recurrent Neural Networks
• Recurrent Neural Networks to automate Quality assessment of Software Requirements
• Knowledge Graphs for Multilingual Language Translation and Generation
• Deep Learning: a new definition of artificial neuron with double weight
• Integrating Approaches to Word Representation
• A language processing algorithm for predicting tactical solutions to an operational planning problem under uncertainty
• Integrating Regular Expressions with Neural Networks via DFA
• CrossedWires: A Dataset of Syntactically Equivalent but Semantically Disparate Deep Learning Models
• Exposure Bias versus Self-Recovery: Are Distortions Really Incremental for Autoregressive Text Generation?
• Transform-Based Feature Map Compression for CNN Inference