Feed-Forward Networks with Attention Can Solve Some Long-Term Memory Problems

Abstract

We propose a simplified model of attention which is applicable to feed-forward neural networks and demonstrate that the resulting model can solve the synthetic "addition" and "multiplication" long-term memory problems for sequence lengths which are both longer and more widely varying than the best published results for these tasks.

References in corpus (10)

• Adam: A Method for Stochastic Optimization
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• On the difficulty of training Recurrent Neural Networks
• Theano: new features and speed improvements
• A Simple Way to Initialize Recurrent Networks of Rectified Linear Units
• Listen, Attend and Spell
• Unitary Evolution Recurrent Neural Networks
• Learning Longer Memory in Recurrent Neural Networks
• Neural Turing Machines
• Regularizing RNNs by Stabilizing Activations

Cited by in corpus (47)

• Attention-based Deep Multiple Instance Learning
• Attentive Statistics Pooling for Deep Speaker Embedding
• Whole Slide Images based Cancer Survival Prediction using Attention Guided Deep Multiple Instance Learning Networks
• AUTSL: A Large Scale Multi-modal Turkish Sign Language Dataset and Baseline Methods
• Robust Attentional Aggregation of Deep Feature Sets for Multi-view 3D Reconstruction
• Deep Learning with Dynamic Computation Graphs
• Weakly Supervised Attention-based Models Using Activation Maps for Citrus Mite and Insect Pest Classification
• Monotonic Chunkwise Attention
• Skeleton Aware Multi-modal Sign Language Recognition
• Attention Back-end for Automatic Speaker Verification with Multiple Enrollment Utterances
• PS-DeVCEM: Pathology-sensitive deep learning model for video capsule endoscopy based on weakly labeled data
• Empirical Analysis of Multi-Task Learning for Reducing Model Bias in Toxic Comment Detection
• On the Importance of Attention in Meta-Learning for Few-Shot Text Classification
• Classifying medical notes into standard disease codes using Machine Learning
• An Attention Mechanism for Musical Instrument Recognition
• Platoon trajectories generation: A unidirectional interconnected LSTM-based car following model
• Are the Multilingual Models Better? Improving Czech Sentiment with Transformers
• Clustering Text Using Attention
• Adaptive pooling operators for weakly labeled sound event detection
• Attention-Based Learning on Molecular Ensembles
• Listen Attentively, and Spell Once: Whole Sentence Generation via a Non-Autoregressive Architecture for Low-Latency Speech Recognition
• Helping each Other: A Framework for Customer-to-Customer Suggestion Mining using a Semi-supervised Deep Neural Network
• Extracting Problem and Method Sentence from Scientific Papers: A Context-enhanced Transformer Using Formulaic Expression Desensitization
• Focusing on What is Relevant: Time-Series Learning and Understanding using Attention
• Singing Voice Conversion with Disentangled Representations of Singer and Vocal Technique Using Variational Autoencoders
• Spoofing-Aware Attention based ASV Back-end with Multiple Enrollment Utterances and a Sampling Strategy for the SASV Challenge 2022
• Attention Fusion Networks: Combining Behavior and E-mail Content to Improve Customer Support
• Modeling and Predicting Popularity Dynamics via Deep Learning Attention Mechanism
• Character-Based Text Classification using Top Down Semantic Model for Sentence Representation
• Leveraging Multi-domain, Heterogeneous Data using Deep Multitask Learning for Hate Speech Detection
• Cross-scale Attention Model for Acoustic Event Classification
• DynGraph2Seq: Dynamic-Graph-to-Sequence Interpretable Learning for Health Stage Prediction in Online Health Forums
• A Domain Generalization Perspective on Listwise Context Modeling
• BCFNet: A Balanced Collaborative Filtering Network with Attention Mechanism
• Foresee: Attentive Future Projections of Chaotic Road Environments with Online Training
• Exploiting Review Neighbors for Contextualized Helpfulness Prediction
• Improving Interpretability for Computer-aided Diagnosis tools on Whole Slide Imaging with Multiple Instance Learning and Gradient-based Explanations
• Multi-Label Annotation of Chest Abdomen Pelvis Computed Tomography Text Reports Using Deep Learning
• Leveraging Frequent Query Substructures to Generate Formal Queries for Complex Question Answering
• Linking Social Media Posts to News with Siamese Transformers
• Residual Memory Networks: Feed-forward approach to learn long temporal dependencies
• Pedestrian Tracking with Gated Recurrent Units and Attention Mechanisms
• Learning to Reconstruct and Segment 3D Objects
• Semantic Relatedness Based Re-ranker for Text Spotting
• Zero-shot Singing Technique Conversion
• Using holistic event information in the trigger
• Learning to Focus when Ranking Answers