On the difficulty of training Recurrent Neural Networks

Abstract

There are two widely known issues with properly training Recurrent Neural Networks, the vanishing and the exploding gradient problems detailed in Bengio et al. (1994). In this paper we attempt to improve the understanding of the underlying issues by exploring these problems from an analytical, a geometric and a dynamical systems perspective. Our analysis is used to justify a simple yet effective solution. We propose a gradient norm clipping strategy to deal with exploding gradients and a soft constraint for the vanishing gradients problem. We validate empirically our hypothesis and proposed solutions in the experimental section.

Improved description of the exploding gradient problem and description and analysis of the vanishing gradient problem

References in corpus (3)

• Theano: new features and speed improvements
• Modeling Temporal Dependencies in High-Dimensional Sequences: Application to Polyphonic Music Generation and Transcription
• Advances in Optimizing Recurrent Networks

Cited by in corpus (636)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Deep Learning in Neural Networks: An Overview
• Sequence to Sequence Learning with Neural Networks
• Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
• Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Network
• Bootstrap your own latent: A new approach to self-supervised Learning
• Deep learning for time series classification: a review
• Character-level Convolutional Networks for Text Classification
• Deep Convolutional Neural Network for Inverse Problems in Imaging
• Conditional Random Fields as Recurrent Neural Networks
• Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
• A Critical Review of Recurrent Neural Networks for Sequence Learning
• DeepSleepNet: a Model for Automatic Sleep Stage Scoring based on Raw Single-Channel EEG
• End-to-End Training of Deep Visuomotor Policies
• Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting
• Deeply-Supervised Nets
• Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
• Exploring the Limits of Language Modeling
• Visualizing and Understanding Recurrent Networks
• A Survey and Critique of Multiagent Deep Reinforcement Learning
• Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges
• Efficiently Modeling Long Sequences with Structured State Spaces
• HDLTex: Hierarchical Deep Learning for Text Classification
• A Physics-Informed Machine Learning Approach for Solving Heat Transfer Equation in Advanced Manufacturing and Engineering Applications
• Linking connectivity, dynamics and computations in low-rank recurrent neural networks
• Joint entity recognition and relation extraction as a multi-head selection problem
• Can Deep Learning Outperform Modern Commercial CT Image Reconstruction Methods?
• Light Gated Recurrent Units for Speech Recognition
• End-to-end Continuous Speech Recognition using Attention-based Recurrent NN: First Results
• Artificial neural networks for neuroscientists: A primer
• Deep Learning for Time Series Forecasting: Tutorial and Literature Survey
• Learning with Pseudo-Ensembles
• Modeling the Dynamics of PDE Systems with Physics-Constrained Deep Auto-Regressive Networks
• Learning Continuous Control Policies by Stochastic Value Gradients
• Selective Encoding for Abstractive Sentence Summarization
• Working Memory Connections for LSTM
• An overview and comparative analysis of Recurrent Neural Networks for Short Term Load Forecasting
• Feed-Forward Networks with Attention Can Solve Some Long-Term Memory Problems
• High-Performance Large-Scale Image Recognition Without Normalization
• The Principles of Deep Learning Theory
• ReNet: A Recurrent Neural Network Based Alternative to Convolutional Networks
• Device Placement Optimization with Reinforcement Learning
• Distributive Dynamic Spectrum Access through Deep Reinforcement Learning: A Reservoir Computing Based Approach
• End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF
• A Gated Recurrent Unit Approach to Bitcoin Price Prediction
• CNN-RNN: A Unified Framework for Multi-label Image Classification
• Gated Graph Recurrent Neural Networks
• On Explaining the Surprising Success of Reservoir Computing Forecaster of Chaos? The Universal Machine Learning Dynamical System with Contrasts to VAR and DMD
• Long Short-Term Memory-Networks for Machine Reading
• Deep learning of dynamics and signal-noise decomposition with time-stepping constraints
• Pix2Vox++: Multi-scale Context-aware 3D Object Reconstruction from Single and Multiple Images
• On Extended Long Short-term Memory and Dependent Bidirectional Recurrent Neural Network
• Learning Constitutive Relations using Symmetric Positive Definite Neural Networks
• Patient2Vec: A Personalized Interpretable Deep Representation of the Longitudinal Electronic Health Record
• The NLP Cookbook: Modern Recipes for Transformer based Deep Learning Architectures
• How Does Learning Rate Decay Help Modern Neural Networks?
• Modeling Human Motion with Quaternion-based Neural Networks
• HiPPO: Recurrent Memory with Optimal Polynomial Projections
• Deep Learning in fNIRS: A review
• Causality Extraction based on Self-Attentive BiLSTM-CRF with Transferred Embeddings
• Gradient Descent Learns Linear Dynamical Systems
• Ensemble Kalman Inversion: A Derivative-Free Technique For Machine Learning Tasks
• Exploiting Cognitive Structure for Adaptive Learning
• Learning the PE Header, Malware Detection with Minimal Domain Knowledge
• CT-image Super Resolution Using 3D Convolutional Neural Network
• Explainable Outfit Recommendation with Joint Outfit Matching and Comment Generation
• Optical Reservoir Computing using multiple light scattering for chaotic systems prediction
• Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNN
• Blocks and Fuel: Frameworks for deep learning
• DialogWAE: Multimodal Response Generation with Conditional Wasserstein Auto-Encoder
• Towards Universal Paraphrastic Sentence Embeddings
• Machine learning and AI-based approaches for bioactive ligand discovery and GPCR-ligand recognition
• Deep Learning of Representations: Looking Forward
• Exploring the Role of Visual Content in Fake News Detection
• BaitWatcher: A lightweight web interface for the detection of incongruent news headlines
• Full-Capacity Unitary Recurrent Neural Networks
• Discover Your Social Identity from What You Tweet: a Content Based Approach
• Review: Deep Learning in Electron Microscopy
• Neural Networks for Modeling and Control of Particle Accelerators
• Machine learning with neural networks
• FastGRNN: A Fast, Accurate, Stable and Tiny Kilobyte Sized Gated Recurrent Neural Network
• Learning Deep Transformer Models for Machine Translation
• Hierarchy-based Image Embeddings for Semantic Image Retrieval
• On Recurrent Neural Networks for learning-based control: recent results and ideas for future developments
• Brain4Cars: Car That Knows Before You Do via Sensory-Fusion Deep Learning Architecture
• Stable Recurrent Models
• R-Transformer: Recurrent Neural Network Enhanced Transformer
• Characterizing Driving Styles with Deep Learning
• Learning to dance: A graph convolutional adversarial network to generate realistic dance motions from audio
• The Non-IID Data Quagmire of Decentralized Machine Learning
• Machine and Deep Learning for IoT Security and Privacy: Applications, Challenges, and Future Directions
• Meta-Learning Update Rules for Unsupervised Representation Learning
• Structure-Infused Copy Mechanisms for Abstractive Summarization
• ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training
• Why gradient clipping accelerates training: A theoretical justification for adaptivity
• Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layers
• DeepCare: A Deep Dynamic Memory Model for Predictive Medicine
• Combining Neural Networks and Log-linear Models to Improve Relation Extraction
• Fast large-scale optimization by unifying stochastic gradient and quasi-Newton methods
• Music transcription modelling and composition using deep learning
• Efficient and Robust Parallel DNN Training through Model Parallelism on Multi-GPU Platform
• A Review of Hidden Markov Models and Recurrent Neural Networks for Event Detection and Localization in Biomedical Signals
• Differential Privacy Has Disparate Impact on Model Accuracy
• Towards Safer Smart Contracts: A Sequence Learning Approach to Detecting Security Threats
• Constraint-Aware Neural Networks for Riemann Problems
• Developing an Unsupervised Real-time Anomaly Detection Scheme for Time Series with Multi-seasonality
• Neural Abstractive Text Summarization with Sequence-to-Sequence Models
• Determination of the edge of criticality in echo state networks through Fisher information maximization
• Deep Learning with Low Precision by Half-wave Gaussian Quantization
• Learning for Detection: MIMO-OFDM Symbol Detection through Downlink Pilots
• Improving performance of recurrent neural network with relu nonlinearity
• Document Context Language Models
• Machine Learning for Prediction with Missing Dynamics
• Variational Temporal Deep Generative Model for Radar HRRP Target Recognition
• Multi-Task Learning for Sequence Tagging: An Empirical Study
• Deep Temporal Sigmoid Belief Networks for Sequence Modeling
• ClariNet: Parallel Wave Generation in End-to-End Text-to-Speech
• Improving Outfit Recommendation with Co-supervision of Fashion Generation
• Analysis of Railway Accidents' Narratives Using Deep Learning
• Sparsity through evolutionary pruning prevents neuronal networks from overfitting
• Supervised Speech Separation Based on Deep Learning: An Overview
• Neural Networks Fail to Learn Periodic Functions and How to Fix It
• Asymmetric Residual Neural Network for Accurate Human Activity Recognition
• Learning Long-Term Dependencies in Irregularly-Sampled Time Series
• Deep Recurrent Convolutional Networks for Video-based Person Re-identification: An End-to-End Approach
• Recurrent Batch Normalization
• On the smoothness of nonlinear system identification
• Neural Network Augmented Physics Models for Systems with Partially Unknown Dynamics: Application to Slider-Crank Mechanism
• Adaptive Input Representations for Neural Language Modeling
• Normalization Techniques in Training DNNs: Methodology, Analysis and Application
• On End-to-End Program Generation from User Intention by Deep Neural Networks
• ServeNet: A Deep Neural Network for Web Services Classification
• Adversarial Examples in Modern Machine Learning: A Review
• Self-Adaptive Hierarchical Sentence Model
• Interpreting recurrent neural networks behaviour via excitable network attractors
• A Hybrid Spatial-temporal Deep Learning Architecture for Lane Detection
• QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answering
• Lipschitz Recurrent Neural Networks
• End-to-End Tracking and Semantic Segmentation Using Recurrent Neural Networks
• Unit Tests for Stochastic Optimization
• Quantitative Fine-Grained Human Evaluation of Machine Translation Systems: a Case Study on English to Croatian
• Spatio-Temporal Point Process for Multiple Object Tracking
• Beyond Convexity: Stochastic Quasi-Convex Optimization
• Augmenting Self-attention with Persistent Memory
• Multichannel End-to-end Speech Recognition
• EEG-Based Emotion Recognition Using Regularized Graph Neural Networks
• Adversarial Examples - A Complete Characterisation of the Phenomenon
• Diverse Human Motion Prediction Guided by Multi-Level Spatial-Temporal Anchors
• Recommender systems based on graph embedding techniques: A comprehensive review
• A Reputation Mechanism Is All You Need: Collaborative Fairness and Adversarial Robustness in Federated Learning
• List-Mode PET Image Reconstruction Using Deep Image Prior
• Neural ranking models for document retrieval
• Implicit Distortion and Fertility Models for Attention-based Encoder-Decoder NMT Model
• Predictive-Corrective Networks for Action Detection
• Universality and individuality in neural dynamics across large populations of recurrent networks
• Robust Forecasting using Predictive Generalized Synchronization in Reservoir Computing
• Semantic Image Networks for Human Action Recognition
• Reinforced Genetic Algorithm Learning for Optimizing Computation Graphs
• Implicit Language Model in LSTM for OCR
• An Improvement of Data Classification Using Random Multimodel Deep Learning (RMDL)
• A Comparison of Machine Learning Algorithms for the Surveillance of Autism Spectrum Disorder
• Recurrent Neural Networks with External Memory for Language Understanding
• Toward Interpretable Music Tagging with Self-Attention
• Deep Independently Recurrent Neural Network (IndRNN)
• Multi-Graph Transformer for Free-Hand Sketch Recognition
• Letter-Based Speech Recognition with Gated ConvNets
• Zero-Resource Translation with Multi-Lingual Neural Machine Translation
• On the Use of Neural Networks for Full Waveform Inversion
• Adaptive Checkpoint Adjoint Method for Gradient Estimation in Neural ODE
• An attentive neural architecture for joint segmentation and parsing and its application to real estate ads
• Gradient Centralization: A New Optimization Technique for Deep Neural Networks
• Tensor Normalization and Full Distribution Training
• Cryptocurrency Trading: A Comprehensive Survey
• Regularization and nonlinearities for neural language models: when are they needed?
• Position-Aware Self-Attention based Neural Sequence Labeling
• Deep Distributional Time Series Models and the Probabilistic Forecasting of Intraday Electricity Prices
• Forecasting Sequential Data using Consistent Koopman Autoencoders
• A Kronecker-factored approximate Fisher matrix for convolution layers
• Real or Fake? Learning to Discriminate Machine from Human Generated Text
• Predicting purchasing intent: Automatic Feature Learning using Recurrent Neural Networks
• L2-Nonexpansive Neural Networks
• Gated Recurrent Neural Tensor Network
• End-to-end Named Entity Recognition and Relation Extraction using Pre-trained Language Models
• Efficient implementations of echo state network cross-validation
• Why Spectral Normalization Stabilizes GANs: Analysis and Improvements
• Break-It-Fix-It: Unsupervised Learning for Program Repair
• Target Detection and Segmentation in Circular-Scan Synthetic-Aperture-Sonar Images using Semi-Supervised Convolutional Encoder-Decoders
• Recurrent Neural Network-based Internal Model Control design for stable nonlinear systems
• Who Needs Words? Lexicon-Free Speech Recognition
• CASTLE: Regularization via Auxiliary Causal Graph Discovery
• IoT Data Analytics Using Deep Learning
• Controllable Abstractive Summarization
• A bio-inspired bistable recurrent cell allows for long-lasting memory
• The Echo Index and multistability in input-driven recurrent neural networks
• DP-BART for Privatized Text Rewriting under Local Differential Privacy
• Como funciona o Deep Learning
• TF-Coder: Program Synthesis for Tensor Manipulations
• Random Error Sampling-based Recurrent Neural Network Architecture Optimization
• On Extractive and Abstractive Neural Document Summarization with Transformer Language Models
• Dynamical Isometry and a Mean Field Theory of LSTMs and GRUs
• Learning with hidden variables
• Bayesian Neural Architecture Search using A Training-Free Performance Metric
• Structure-Tags Improve Text Classification for Scholarly Document Quality Prediction
• What does it mean to understand a neural network?
• Recurrent Memory Networks for Language Modeling
• Conducting Credit Assignment by Aligning Local Representations
• Adaptive Learning Rate Clipping Stabilizes Learning
• Analysing Deep Reinforcement Learning Agents Trained with Domain Randomisation
• Fractional Underdamped Langevin Dynamics: Retargeting SGD with Momentum under Heavy-Tailed Gradient Noise
• Correspondence between neuroevolution and gradient descent
• Adaptively Truncating Backpropagation Through Time to Control Gradient Bias
• Anchor Diffusion for Unsupervised Video Object Segmentation
• No Padding Please: Efficient Neural Handwriting Recognition
• AngularGrad: A New Optimization Technique for Angular Convergence of Convolutional Neural Networks
• General Value Function Networks
• Efficient Optimization of Echo State Networks for Time Series Datasets
• Sigsoftmax: Reanalysis of the Softmax Bottleneck
• Prompt Agnostic Essay Scorer: A Domain Generalization Approach to Cross-prompt Automated Essay Scoring
• Discriminative Nearest Neighbor Few-Shot Intent Detection by Transferring Natural Language Inference
• Exploring Transformers in Natural Language Generation: GPT, BERT, and XLNet
• Annotation Order Matters: Recurrent Image Annotator for Arbitrary Length Image Tagging
• SNODE: Spectral Discretization of Neural ODEs for System Identification
• Learning to Parse and Translate Improves Neural Machine Translation
• Variational Recurrent Models for Solving Partially Observable Control Tasks
• Non-normal Recurrent Neural Network (nnRNN): learning long time dependencies while improving expressivity with transient dynamics
• Stochastic Shared Embeddings: Data-driven Regularization of Embedding Layers
• A Deep Generative Model of Speech Complex Spectrograms
• A Signal Propagation Perspective for Pruning Neural Networks at Initialization
• A Modular Task-oriented Dialogue System Using a Neural Mixture-of-Experts
• Probabilistic Forecasting with Temporal Convolutional Neural Network
• Deep Complex Networks for Protocol-Agnostic Radio Frequency Device Fingerprinting in the Wild
• LSTM Neural Networks: Input to State Stability and Probabilistic Safety Verification
• Deep Hamiltonian networks based on symplectic integrators
• GNPassGAN: Improved Generative Adversarial Networks For Trawling Offline Password Guessing
• Transfer Learning for Clinical Time Series Analysis using Recurrent Neural Networks
• Modeling citation worthiness by using attention-based bidirectional long short-term memory networks and interpretable models
• The challenge of simultaneous object detection and pose estimation: a comparative study
• Residual Energy-Based Models for Text
• Toward Mention Detection Robustness with Recurrent Neural Networks
• Investigating gated recurrent neural networks for speech synthesis
• Bidirectional Recurrent Neural Networks as Generative Models - Reconstructing Gaps in Time Series
• Discriminative Sentence Modeling for Story Ending Prediction
• On the distance between two neural networks and the stability of learning
• Language models and Automated Essay Scoring
• Scene Memory Transformer for Embodied Agents in Long-Horizon Tasks
• ConceptBeam: Concept Driven Target Speech Extraction
• Technical Considerations for Semantic Segmentation in MRI using Convolutional Neural Networks
• Prediction of the motion of chest internal points using a recurrent neural network trained with real-time recurrent learning for latency compensation in lung cancer radiotherapy
• Keep the Gradients Flowing: Using Gradient Flow to Study Sparse Network Optimization
• Utterance-level Dialogue Understanding: An Empirical Study
• The Statistical Recurrent Unit
• DAG-Recurrent Neural Networks For Scene Labeling
• Coupled Oscillatory Recurrent Neural Network (coRNN): An accurate and (gradient) stable architecture for learning long time dependencies
• Estimation and Control of Motor Core Temperature with Online Learning of Thermal Model Parameters: Application to Musculoskeletal Humanoids
• Stability and Convergence of Stochastic Gradient Clipping: Beyond Lipschitz Continuity and Smoothness
• A Novel Approach Based Deep RNN Using Hybrid NARX-LSTM Model For Solar Power Forecasting
• A Multi-dimensional Deep Structured State Space Approach to Speech Enhancement Using Small-footprint Models
• Smooth Monotone Stochastic Variational Inequalities and Saddle Point Problems: A Survey
• Neural Sign Language Translation based on Human Keypoint Estimation
• Can Unconditional Language Models Recover Arbitrary Sentences?
• A Fully Differentiable Beam Search Decoder
• Strongly-Typed Recurrent Neural Networks
• Heavy Ball Neural Ordinary Differential Equations
• Training Algorithm Matters for the Performance of Neural Network Potential: A Case Study of Adam and the Kalman Filter Optimizers
• Model Reduction with Memory and the Machine Learning of Dynamical Systems
• One-Shot Pruning of Recurrent Neural Networks by Jacobian Spectrum Evaluation
• Learning to Remember More with Less Memorization
• Assessment of Data Consistency through Cascades of Independently Recurrent Inference Machines for fast and robust accelerated MRI reconstruction
• Differentiable Programming of Isometric Tensor Networks
• Image compression optimized for 3D reconstruction by utilizing deep neural networks
• Maximizing Mutual Information for Tacotron
• Memorized Sparse Backpropagation
• AlignTransformer: Hierarchical Alignment of Visual Regions and Disease Tags for Medical Report Generation
• Learning Input and Recurrent Weight Matrices in Echo State Networks
• Robust Multilingual Part-of-Speech Tagging via Adversarial Training
• Effects of the Nonlinearity in Activation Functions on the Performance of Deep Learning Models
• CKConv: Continuous Kernel Convolution For Sequential Data
• Can SGD Learn Recurrent Neural Networks with Provable Generalization?
• Reservoir Computing meets Recurrent Kernels and Structured Transforms
• MART: Memory-Augmented Recurrent Transformer for Coherent Video Paragraph Captioning
• Calligraphic Stylisation Learning with a Physiologically Plausible Model of Movement and Recurrent Neural Networks
• Deep Learning Based Energy Disaggregation and On/Off Detection of Household Appliances
• Seizure Prediction Using Bidirectional LSTM
• End-to-end Recurrent Neural Network Models for Vietnamese Named Entity Recognition: Word-level vs. Character-level
• Demystifying Deep Learning in Predictive Spatio-Temporal Analytics: An Information-Theoretic Framework
• Low-Cost Recurrent Neural Network Expected Performance Evaluation
• Prediction of the Position of External Markers Using a Recurrent Neural Network Trained With Unbiased Online Recurrent Optimization for Safe Lung Cancer Radiotherapy
• Gated Embeddings in End-to-End Speech Recognition for Conversational-Context Fusion
• Sequential Deep Learning for Credit Risk Monitoring with Tabular Financial Data
• Toward a Period-Specific Optimized Neural Network for OCR Error Correction of Historical Hebrew Texts
• Neuroevolution on the Edge of Chaos
• A Multidimensional Graph Fourier Transformation Neural Network for Vehicle Trajectory Prediction
• NAC-TCN: Temporal Convolutional Networks with Causal Dilated Neighborhood Attention for Emotion Understanding
• COVID-19 growth prediction using multivariate long short term memory
• Magnetic Resonance Fingerprinting Reconstruction Using Recurrent Neural Networks
• A Weight Initialization Based on the Linear Product Structure for Neural Networks
• UnICORNN: A recurrent model for learning very long time dependencies
• TLDR: Token Loss Dynamic Reweighting for Reducing Repetitive Utterance Generation
• MALI: A memory efficient and reverse accurate integrator for Neural ODEs
• Reversible Recursive Instance-level Object Segmentation
• Effect of Architectures and Training Methods on the Performance of Learned Video Frame Prediction
• Interpretable Adversarial Training for Text
• Stochastic Optimization with Heavy-Tailed Noise via Accelerated Gradient Clipping
• Enabling High Performance Debugging for Variational Quantum Algorithms using Compressed Sensing
• GCDT: A Global Context Enhanced Deep Transition Architecture for Sequence Labeling
• DeepSense: Enabling Carrier Sense in Low-Power Wide Area Networks Using Deep Learning
• Dissecting Deep Neural Networks
• Modeling Personalized Item Frequency Information for Next-basket Recommendation
• Improved Analysis of Clipping Algorithms for Non-convex Optimization
• Improving Multi-Word Entity Recognition for Biomedical Texts
• Norm-preserving Orthogonal Permutation Linear Unit Activation Functions (OPLU)
• Stochastic gradient descent methods for estimation with large data sets
• Incorporating External POS Tagger for Punctuation Restoration
• Implementing a Bayes Filter in a Neural Circuit: The Case of Unknown Stimulus Dynamics
• Connectionist-Symbolic Machine Intelligence using Cellular Automata based Reservoir-Hyperdimensional Computing
• GASL: Guided Attention for Sparsity Learning in Deep Neural Networks
• Per-Tensor Fixed-Point Quantization of the Back-Propagation Algorithm
• Echo State Neural Machine Translation
• end-to-end training of a large vocabulary end-to-end speech recognition system
• Machine learning-based multiscale constitutive modelling: Development and application to dual-porosity mass transfer
• Learning Convex Optimization Control Policies
• MOrdReD: Memory-based Ordinal Regression Deep Neural Networks for Time Series Forecasting
• Bag-of-Words as Target for Neural Machine Translation
• Improving Generative Imagination in Object-Centric World Models
• Implementing Neural Turing Machines
• End-to-end Speech Recognition with Word-based RNN Language Models
• Natural Language Generation with Neural Variational Models
• A Primal-Dual Method for Training Recurrent Neural Networks Constrained by the Echo-State Property
• SwitchNet: a neural network model for forward and inverse scattering problems
• CORAL8: Concurrent Object Regression for Area Localization in Medical Image Panels
• The Neural Moving Average Model for Scalable Variational Inference of State Space Models
• Learning Deep Neural Network Policies with Continuous Memory States
• Feedforward and Recurrent Neural Networks Backward Propagation and Hessian in Matrix Form
• Personalized Transformer for Explainable Recommendation
• Leveraging End-to-End Speech Recognition with Neural Architecture Search
• Gating creates slow modes and controls phase-space complexity in GRUs and LSTMs
• Visual Depth Mapping from Monocular Images using Recurrent Convolutional Neural Networks
• EmoTwiCS: A Corpus for Modelling Emotion Trajectories in Dutch Customer Service Dialogues on Twitter
• Evading Curse of Dimensionality in Unconstrained Private GLMs via Private Gradient Descent
• Improving Explainable Recommendations with Synthetic Reviews
• Disentangling Adaptive Gradient Methods from Learning Rates
• Context Aware Machine Learning
• Practical Processing of Mobile Sensor Data for Continual Deep Learning Predictions
• A clustering approach to time series forecasting using neural networks: A comparative study on distance-based vs. feature-based clustering methods
• Recurrent Models for Auditory Attention in Multi-Microphone Distance Speech Recognition
• The Nonlinearity Coefficient - Predicting Generalization in Deep Neural Networks
• ArTIST: Autoregressive Trajectory Inpainting and Scoring for Tracking
• Detecting and Classifying Malevolent Dialogue Responses: Taxonomy, Data and Methodology
• Do Sentence Interactions Matter? Leveraging Sentence Level Representations for Fake News Classification
• Orthogonal Convolutional Neural Networks
• Convergence Analysis of a Momentum Algorithm with Adaptive Step Size for Non Convex Optimization
• Sensitivity - Local Index to Control Chaoticity or Gradient Globally -
• Initializing LSTM internal states via manifold learning
• A Tutorial about Random Neural Networks in Supervised Learning
• Learning Variations in Human Motion via Mix-and-Match Perturbation
• Tricks from Deep Learning
• On the Curse of Memory in Recurrent Neural Networks: Approximation and Optimization Analysis
• Learning to Take Directions One Step at a Time
• WildMix Dataset and Spectro-Temporal Transformer Model for Monoaural Audio Source Separation
• Stochastic Training is Not Necessary for Generalization
• Tensor Decomposition for Compressing Recurrent Neural Network
• Learning Sparse Wavelet Representations
• Countering Noisy Labels By Learning From Auxiliary Clean Labels
• Beyond BLEU: Training Neural Machine Translation with Semantic Similarity
• Constraint-Based Regularization of Neural Networks
• Emoji-Powered Representation Learning for Cross-Lingual Sentiment Classification
• Understanding Recurrent Neural Networks Using Nonequilibrium Response Theory
• The Ant Swarm Neuro-Evolution Procedure for Optimizing Recurrent Networks
• A Loss Curvature Perspective on Training Instability in Deep Learning
• Spike-based computation using classical recurrent neural networks
• Wireless Localisation in WiFi using Novel Deep Architectures
• ECAS-ML: Edge Computing Assisted Adaptation Scheme with Machine Learning for HTTP Adaptive Streaming
• Noisy Recurrent Neural Networks
• Domain-Specific Priors and Meta Learning for Few-Shot First-Person Action Recognition
• Recurrent Neural Networks for P300-based BCI
• Exploring spectro-temporal features in end-to-end convolutional neural networks
• Spiking Neural Predictive Coding for Continual Learning from Data Streams
• exploRNN: Understanding Recurrent Neural Networks through Visual Exploration
• Symplectic Adjoint Method for Exact Gradient of Neural ODE with Minimal Memory
• Training Stronger Baselines for Learning to Optimize
• DeepSITH: Efficient Learning via Decomposition of What and When Across Time Scales
• Ab-Initio Potential Energy Surfaces by Pairing GNNs with Neural Wave Functions
• MomentumRNN: Integrating Momentum into Recurrent Neural Networks
• Untangling tradeoffs between recurrence and self-attention in neural networks
• Romanian Speech Recognition Experiments from the ROBIN Project
• Leveraging Deep Learning for Time Series Extrinsic Regression in predicting photometric metallicity of Fundamental-mode RR Lyrae Stars
• A Visual Communication Map for Multi-Agent Deep Reinforcement Learning
• Large-Scale Gradient-Free Deep Learning with Recursive Local Representation Alignment
• Interplay of Traditional Methods and Machine Learning Algorithms for Tagging Boosted Objects
• Do Neural Networks for Segmentation Understand Insideness?
• CM-Net: A Novel Collaborative Memory Network for Spoken Language Understanding
• Automated Polysomnography Analysis for Detection of Non-Apneic and Non-Hypopneic Arousals using Feature Engineering and a Bidirectional LSTM Network
• Auxiliary Task Update Decomposition: The Good, The Bad and The Neutral
• A Lightweight Structure Aimed to Utilize Spatial Correlation for Sparse-View CT Reconstruction
• A Two-branch Neural Network for Non-homogeneous Dehazing via Ensemble Learning
• Benchmarking Deep Sequential Models on Volatility Predictions for Financial Time Series
• Generation of Realistic Cloud Access Times for Mobile Application Testing using Transfer Learning
• Preventing Posterior Collapse with Levenshtein Variational Autoencoder
• Deep Loopy Neural Network Model for Graph Structured Data Representation Learning
• Retrospective and Prospective Mixture-of-Generators for Task-oriented Dialogue Response Generation
• Audio-Linguistic Embeddings for Spoken Sentences
• OSVNet: Convolutional Siamese Network for Writer Independent Online Signature Verification
• Arabic aspect based sentiment analysis using bidirectional GRU based models
• The Nonlinearity Coefficient - A Practical Guide to Neural Architecture Design
• DDPNOpt: Differential Dynamic Programming Neural Optimizer
• Reinforced stochastic gradient descent for deep neural network learning
• Meta Continual Learning via Dynamic Programming
• Reduction of rain-induced errors for wind speed estimation on SAR observations using convolutional neural networks
• Multiresolution Transformer Networks: Recurrence is Not Essential for Modeling Hierarchical Structure
• Simplified Gating in Long Short-term Memory (LSTM) Recurrent Neural Networks
• Challenges for cognitive decoding using deep learning methods
• Contracting Implicit Recurrent Neural Networks: Stable Models with Improved Trainability
• Scenario optimization for optimal training of Echo State Networks
• A Hierarchical Recurrent Encoder-Decoder For Generative Context-Aware Query Suggestion
• GraphTCN: Spatio-Temporal Interaction Modeling for Human Trajectory Prediction
• A Neural Solver for Variational Problems on CAD Geometries with Application to Electric Machine Simulation
• Variable-Rate Deep Image Compression through Spatially-Adaptive Feature Transform
• Unsupervised Domain Adaptation for Multispectral Pedestrian Detection
• Extreme Relative Pose Network under Hybrid Representations
• Improving Long Handwritten Text Line Recognition with Convolutional Multi-way Associative Memory
• Equilibrated Recurrent Neural Network: Neuronal Time-Delayed Self-Feedback Improves Accuracy and Stability
• Using stigmergy as a computational memory in the design of recurrent neural networks
• Learning to Denoise and Decode: A Novel Residual Neural Network Decoder for Polar Codes
• TopicEq: A Joint Topic and Mathematical Equation Model for Scientific Texts
• Cross-Attention End-to-End ASR for Two-Party Conversations
• Probabilistic Modelling of Morphologically Rich Languages
• Learned-Norm Pooling for Deep Feedforward and Recurrent Neural Networks
• Instance-Based Learning of Span Representations: A Case Study through Named Entity Recognition
• Wide Neural Networks Forget Less Catastrophically
• Belief-Grounded Networks for Accelerated Robot Learning under Partial Observability
• Stepwise Model Selection for Sequence Prediction via Deep Kernel Learning
• An In-depth Summary of Recent Artificial Intelligence Applications in Drug Design
• Deep Autotuner: a Pitch Correcting Network for Singing Performances
• ORGaNICs: A Theory of Working Memory in Brains and Machines
• Efficient Neural Network Implementation with Quadratic Neuron
• Interpreting Depression From Question-wise Long-term Video Recording of SDS Evaluation
• Integrating Dictionary Feature into A Deep Learning Model for Disease Named Entity Recognition
• Refined Gate: A Simple and Effective Gating Mechanism for Recurrent Units
• Compressing LSTM Networks by Matrix Product Operators
• Neural Text Generation with Artificial Negative Examples
• A Dynamically Controlled Recurrent Neural Network for Modeling Dynamical Systems
• Towards Deep Physical Reservoir Computing Through Automatic Task Decomposition And Mapping
• NeuralPDR: Neural Differential Equations as surrogate models for Photodissociation Regions
• S++: A Fast and Deployable Secure-Computation Framework for Privacy-Preserving Neural Network Training
• Customizing Sequence Generation with Multi-Task Dynamical Systems
• State-Denoised Recurrent Neural Networks
• Learning Diverse-Structured Networks for Adversarial Robustness
• Comparative Analysis of the Hidden Markov Model and LSTM: A Simulative Approach
• Learning Recurrent Binary/Ternary Weights
• ECG-DelNet: Delineation of Ambulatory Electrocardiograms with Mixed Quality Labeling Using Neural Networks
• Generalized few-shot transfer learning architecture for modeling the EDFA gain spectrum
• Real-time Forecast Models for TBM Load Parameters Based on Machine Learning Methods
• Shapley Interpretation and Activation in Neural Networks
• Feedback Gradient Descent: Efficient and Stable Optimization with Orthogonality for DNNs
• A non-autonomous equation discovery method for time signal classification
• Attention Fusion Networks: Combining Behavior and E-mail Content to Improve Customer Support
• TandemNet: Distilling Knowledge from Medical Images Using Diagnostic Reports as Optional Semantic References
• An Adaptive Stochastic Nesterov Accelerated Quasi Newton Method for Training RNNs
• EditNTS: An Neural Programmer-Interpreter Model for Sentence Simplification through Explicit Editing
• One Size Does Not Fit All: Multi-Scale, Cascaded RNNs for Radar Classification
• Microsoft AI Challenge India 2018: Learning to Rank Passages for Web Question Answering with Deep Attention Networks
• This Email Could Save Your Life: Introducing the Task of Email Subject Line Generation
• Context Attentive Document Ranking and Query Suggestion
• Linear Range in Gradient Descent
• Enhancing Sindhi Word Segmentation using Subword Representation Learning and Position-aware Self-attention
• Diabetes Mellitus Forecasting Using Population Health Data in Ontario, Canada
• RNNs Evolving on an Equilibrium Manifold: A Panacea for Vanishing and Exploding Gradients?
• Temporal Collaborative Ranking Via Personalized Transformer
• An Empirical Study of the L2-Boost technique with Echo State Networks
• Deep learning for brake squeal: vibration detection, characterization and prediction
• R-FORCE: Robust Learning for Random Recurrent Neural Networks
• Expectigrad: Fast Stochastic Optimization with Robust Convergence Properties
• Operational Calculus for Differentiable Programming
• Specular reflections removal in colposcopic images based on neural networks: Supervised training with no ground truth previous knowledge
• Reliable and Fast Recurrent Neural Network Architecture Optimization
• RATCHET: Medical Transformer for Chest X-ray Diagnosis and Reporting
• Controllable Length Control Neural Encoder-Decoder via Reinforcement Learning
• Training Spiking Neural Networks for Cognitive Tasks: A Versatile Framework Compatible to Various Temporal Codes
• Key Fact as Pivot: A Two-Stage Model for Low Resource Table-to-Text Generation
• Reverse engineering learned optimizers reveals known and novel mechanisms
• Iterative Text-based Editing of Talking-heads Using Neural Retargeting
• Time-Series Event Prediction with Evolutionary State Graph
• Dynamic Key-Value Memory Networks for Knowledge Tracing
• A Differential Geometry Perspective on Orthogonal Recurrent Models
• Neural Networks for Lorenz Map Prediction: A Trip Through Time
• Understanding and Controlling Memory in Recurrent Neural Networks
• On the Memory Mechanism of Tensor-Power Recurrent Models
• On the biological plausibility of orthogonal initialisation for solving gradient instability in deep neural networks
• Controllable Orthogonalization in Training DNNs
• Scaling Law for Recommendation Models: Towards General-purpose User Representations
• Multi-Source Pointer Network for Product Title Summarization
• Hyperparameter optimization with REINFORCE and Transformers
• Practical License Plate Recognition in Unconstrained Surveillance Systems with Adversarial Super-Resolution
• Eigenvalue Normalized Recurrent Neural Networks for Short Term Memory
• A Novel Topology for End-to-end Temporal Classification and Segmentation with Recurrent Neural Network
• Learning Diverse Stochastic Human-Action Generators by Learning Smooth Latent Transitions
• power-law nonlinearity with maximally uniform distribution criterion for improved neural network training in automatic speech recognition
• Towards Deeper Deep Reinforcement Learning with Spectral Normalization
• Design of Capacity-Approaching Low-Density Parity-Check Codes using Recurrent Neural Networks
• Recurrent neural network training with preconditioned stochastic gradient descent
• A Multi-task Two-stream Spatiotemporal Convolutional Neural Network for Convective Storm Nowcasting
• RNN-based Generative Model for Fine-Grained Sketching
• Deep Autotuner: A Data-Driven Approach to Natural-Sounding Pitch Correction for Singing Voice in Karaoke Performances
• A Study on Graph-Structured Recurrent Neural Networks and Sparsification with Application to Epidemic Forecasting
• Learning to Adaptively Scale Recurrent Neural Networks
• Regularizing Recurrent Networks - On Injected Noise and Norm-based Methods
• Memory Matters: Convolutional Recurrent Neural Network for Scene Text Recognition
• BiSET: Bi-directional Selective Encoding with Template for Abstractive Summarization
• Neural Machine Translation: A Review and Survey
• Improving the Long-Range Performance of Gated Graph Neural Networks
• Future Vector Enhanced LSTM Language Model for LVCSR
• A constrained recursion algorithm for batch normalization of tree-sturctured LSTM
• Differentiating through the Fréchet Mean
• Unsupervised Domain Alignment to Mitigate Low Level Dataset Biases
• Contrastive Video Representation Learning via Adversarial Perturbations
• Gradients are Not All You Need
• An Empirical Exploration of Deep Recurrent Connections and Memory Cells Using Neuro-Evolution
• LIP: Learning Instance Propagation for Video Object Segmentation
• End-to-End Speech Recognition with High-Frame-Rate Features Extraction
• Scene-Aware Error Modeling of LiDAR/Visual Odometry for Fusion-based Vehicle Localization
• Mapping high-performance RNNs to in-memory neuromorphic chips
• Long Expressive Memory for Sequence Modeling
• Neural Network-based Virtual Microphone Estimator
• On Architectures for Including Visual Information in Neural Language Models for Image Description
• Detecting Logical Relation In Contract Clauses
• Deep Unitary Convolutional Neural Networks
• Self-regularizing restricted Boltzmann machines
• Stabilizing Deep Tomographic Reconstruction
• Robust Network Slicing: Multi-Agent Policies, Adversarial Attacks, and Defensive Strategies
• Stochastic gradient-free descents
• Machine Learning and System Identification for Estimation in Physical Systems
• LM-Critic: Language Models for Unsupervised Grammatical Error Correction
• Advances in Collaborative Filtering and Ranking
• Polygonal Unadjusted Langevin Algorithms: Creating stable and efficient adaptive algorithms for neural networks
• Depth-Adaptive Graph Recurrent Network for Text Classification
• Sampling-based Gradient Regularization for Capturing Long-Term Dependencies in Recurrent Neural Networks
• Rank Ordered Autoencoders
• Backward Gradient Normalization in Deep Neural Networks
• Learning Question-Guided Video Representation for Multi-Turn Video Question Answering
• Graph Convolution Networks for Probabilistic Modeling of Driving Acceleration
• Box-level Segmentation Supervised Deep Neural Networks for Accurate and Real-time Multispectral Pedestrian Detection
• Learning and Generalization in RNNs
• Approximate Fixed-Points in Recurrent Neural Networks
• Data Augmentation for Cross-Domain Named Entity Recognition
• PyEuroVoc: A Tool for Multilingual Legal Document Classification with EuroVoc Descriptors
• Generative Models for Security: Attacks, Defenses, and Opportunities
• Multi-class Multilingual Classification of Wikipedia Articles Using Extended Named Entity Tag Set
• Using local plasticity rules to train recurrent neural networks
• A Lightweight Recurrent Network for Sequence Modeling
• Memory and attention in deep learning
• HNC: Leveraging Hard Negative Captions towards Models with Fine-Grained Visual-Linguistic Comprehension Capabilities
• Semantic Role Labeling as Syntactic Dependency Parsing
• Empirical Evaluation of A New Approach to Simplifying Long Short-term Memory (LSTM)
• Quasi-random sampling for multivariate distributions via generative neural networks
• "BNN - BN = ?": Training Binary Neural Networks without Batch Normalization
• Deepening Lee-Carter for longevity projections with uncertainty estimation
• A Simple Recurrent Unit with Reduced Tensor Product Representations
• Improving Neuroevolution Using Island Extinction and Repopulation
• End-to-end Speech Recognition with Adaptive Computation Steps
• Learning Various Length Dependence by Dual Recurrent Neural Networks
• Overcoming Overfitting and Large Weight Update Problem in Linear Rectifiers: Thresholded Exponential Rectified Linear Units
• Deep Learning Models for Early Detection and Prediction of the spread of Novel Coronavirus (COVID-19)
• Fractional order graph neural network
• Improving OOV Detection and Resolution with External Language Models in Acoustic-to-Word ASR
• Improving Natural Language Interaction with Robots Using Advice
• Applications of Machine Learning to Optimizing Polyolefin Manufacturing
• Metadata Shaping: Natural Language Annotations for the Tail
• Machine Learning based Post Processing Artifact Reduction in HEVC Intra Coding
• WeightAlign: Normalizing Activations by Weight Alignment
• Acoustic-to-Word Models with Conversational Context Information
• A geometrical analysis of global stability in trained feedback networks
• Depth Enables Long-Term Memory for Recurrent Neural Networks
• Small energy masking for improved neural network training for end-to-end speech recognition
• DeepDrawing: A Deep Learning Approach to Graph Drawing
• Multi-Decoder RNN Autoencoder Based on Variational Bayes Method
• An In-depth Walkthrough on Evolution of Neural Machine Translation
• Deep Learning based Urban Vehicle Trajectory Analytics
• Improving Adversarial Robustness for Free with Snapshot Ensemble
• Learning Longer-term Dependencies via Grouped Distributor Unit
• Transfer Learning for Neural Networks-based Equalizers in Coherent Optical Systems
• RNN Training along Locally Optimal Trajectories via Frank-Wolfe Algorithm
• Pastprop-RNN: improved predictions of the future by correcting the past
• Surgical Instruction Generation with Transformers
• On the security relevance of weights in deep learning
• Longitudinal Pooling & Consistency Regularization to Model Disease Progression from MRIs
• Context-Aware Attentive Knowledge Tracing
• Rapid training of deep neural networks without skip connections or normalization layers using Deep Kernel Shaping
• An Empirical Study on Measuring the Similarity of Sentential Arguments with Language Model Domain Adaptation
• Predicting Coordinated Actuated Traffic Signal Change Times using LSTM Neural Networks
• Continual Prediction of Notification Attendance with Classical and Deep Network Approaches
• Using holistic event information in the trigger
• Instance-Based Neural Dependency Parsing
• Self-Attentive 3D Human Pose and Shape Estimation from Videos
• A Deep Learning Approach for Automotive Radar Interference Mitigation
• The Assistive Multi-Armed Bandit
• Self-supervised Knowledge Distillation Using Singular Value Decomposition
• Iterative evaluation of LSTM cells
• Unsupervised and Supervised Structure Learning for Protein Contact Prediction
• Deep Neural Networks with Short Circuits for Improved Gradient Learning
• Deep Sequence Learning for Video Anticipation: From Discrete and Deterministic to Continuous and Stochastic
• An Experimental Study of Weight Initialization and Weight Inheritance Effects on Neuroevolution
• PipeTune: Pipeline Parallelism of Hyper and System Parameters Tuning for Deep Learning Clusters
• Novel Weight Update Scheme for Hardware Neural Network based on Synaptic Devices Having Abrupt LTP or LTD Characteristics
• Pre-train, Interact, Fine-tune: A Novel Interaction Representation for Text Classification
• Exploiting Invertible Decoders for Unsupervised Sentence Representation Learning
• Sign and Relevance Learning
• Symmetric Dilated Convolution for Surgical Gesture Recognition
• Constructing Gradient Controllable Recurrent Neural Networks Using Hamiltonian Dynamics
• Learning Memory-Dependent Continuous Control from Demonstrations
• Anomaly Detection of Time Series with Smoothness-Inducing Sequential Variational Auto-Encoder
• Motion Generation Considering Situation with Conditional Generative Adversarial Networks for Throwing Robots
• Evolutionary aspects of Reservoir Computing
• Deeply Shared Filter Bases for Parameter-Efficient Convolutional Neural Networks
• A Click Sequence Model for Web Search
• Comparative Code Structure Analysis using Deep Learning for Performance Prediction
• Decoupling Hierarchical Recurrent Neural Networks With Locally Computable Losses
• Multilevel Monte Carlo Variational Inference
• Diversifying Topic-Coherent Response Generation for Natural Multi-turn Conversations
• A Question Type Driven and Copy Loss Enhanced Frameworkfor Answer-Agnostic Neural Question Generation
• Revisit Systematic Generalization via Meaningful Learning
• Attentive Long Short-Term Preference Modeling for Personalized Product Search
• Learning to Transfer Visual Effects from Videos to Images
• Reconstructing unseen modalities and pathology with an efficient Recurrent Inference Machine
• Regularizing Recurrent Neural Networks via Sequence Mixup
• Diagnosis and Analysis of Celiac Disease and Environmental Enteropathy on Biopsy Images using Deep Learning Approaches
• Interpretable Deep Learning for Stock Returns: A Consensus-Bottleneck Asset Pricing Model
• Understanding Feature Selection and Feature Memorization in Recurrent Neural Networks
• Gradient-based Hyperparameter Optimization Over Long Horizons
• Neural network emulator to constrain the high-$z$ IGM thermal state from Lyman-$\alpha$ forest flux auto-correlation function
• Extended Answer and Uncertainty Aware Neural Question Generation
• Water Quality Prediction on a Sigfox-compliant IoT Device: The Road Ahead of WaterS
• Using a Neural Network Classifier to Select Galaxies with the Most Accurate Photometric Redshifts
• Learning to Control using Image Feedback
• Discriminative Video Representation Learning Using Support Vector Classifiers
• Learning Representations from Road Network for End-to-End Urban Growth Simulation
• How to Write Summaries with Patterns? Learning towards Abstractive Summarization through Prototype Editing