Learning to Remember Rare Events

Abstract

Despite recent advances, memory-augmented deep neural networks are still limited when it comes to life-long and one-shot learning, especially in remembering rare events. We present a large-scale life-long memory module for use in deep learning. The module exploits fast nearest-neighbor algorithms for efficiency and thus scales to large memory sizes. Except for the nearest-neighbor query, the module is fully differentiable and trained end-to-end with no extra supervision. It operates in a life-long manner, i.e., without the need to reset it during training. Our memory module can be easily added to any part of a supervised neural network. To show its versatility we add it to a number of networks, from simple convolutional ones tested on image classification to deep sequence-to-sequence and recurrent-convolutional models. In all cases, the enhanced network gains the ability to remember and do life-long one-shot learning. Our module remembers training examples shown many thousands of steps in the past and it can successfully generalize from them. We set new state-of-the-art for one-shot learning on the Omniglot dataset and demonstrate, for the first time, life-long one-shot learning in recurrent neural networks on a large-scale machine translation task.

Conference paper accepted for ICLR'17

References in corpus (7)

• Adam: A Method for Stochastic Optimization
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Large-scale Simple Question Answering with Memory Networks
• The Goldilocks Principle: Reading Children's Books with Explicit Memory Representations
• One-shot Learning with Memory-Augmented Neural Networks
• Hierarchical Memory Networks

Cited by in corpus (58)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Generalizing from a Few Examples: A Survey on Few-Shot Learning
• Deep Reinforcement Learning: An Overview
• Few-Shot Learning with Graph Neural Networks
• A Two-Stage Approach to Few-Shot Learning for Image Recognition
• Bilevel Programming for Hyperparameter Optimization and Meta-Learning
• Deep Meta-Learning: Learning to Learn in the Concept Space
• Neural Episodic Control
• Few-Shot Learning Through an Information Retrieval Lens
• Abstractive Summarization of Reddit Posts with Multi-level Memory Networks
• CrossTransformers: spatially-aware few-shot transfer
• Small Sample Learning in Big Data Era
• Adversarially Robust Few-Shot Learning: A Meta-Learning Approach
• Search Engine Guided Non-Parametric Neural Machine Translation
• Been There, Done That: Meta-Learning with Episodic Recall
• Deep Triplet Ranking Networks for One-Shot Recognition
• Memory Based Online Learning of Deep Representations from Video Streams
• Meta-Graph: Few Shot Link Prediction via Meta Learning
• The Variational Homoencoder: Learning to learn high capacity generative models from few examples
• Regular Polytope Networks
• Concept Learning through Deep Reinforcement Learning with Memory-Augmented Neural Networks
• Fast Parametric Learning with Activation Memorization
• Few-shot Learning with Meta Metric Learners
• Guiding Neural Machine Translation with Retrieved Translation Pieces
• Learning to Learn Variational Semantic Memory
• Learning to Focus: Cascaded Feature Matching Network for Few-shot Image Recognition
• Few-Shot Video Classification via Temporal Alignment
• Visual Question Answering with Memory-Augmented Networks
• Neural Machine Translation with Noisy Lexical Constraints
• A Closer Look at Few-Shot Video Classification: A New Baseline and Benchmark
• Learning to Remember Translation History with a Continuous Cache
• Are Few-Shot Learning Benchmarks too Simple ? Solving them without Task Supervision at Test-Time
• Cross-domain Recommendation via Deep Domain Adaptation
• Adaptive Posterior Learning: few-shot learning with a surprise-based memory module
• Expert Training: Task Hardness Aware Meta-Learning for Few-Shot Classification
• A Simple Cache Model for Image Recognition
• On the Role of Dataset Quality and Heterogeneity in Model Confidence
• Dynamic Memory Induction Networks for Few-Shot Text Classification
• SQUID: Deep Feature In-Painting for Unsupervised Anomaly Detection
• Towards Continuous Domain adaptation for Healthcare
• Compositional Few-Shot Recognition with Primitive Discovery and Enhancing
• Making Good on LSTMs' Unfulfilled Promise
• Large-Scale Historical Watermark Recognition: dataset and a new consistency-based approach
• Meta Learning with Differentiable Closed-form Solver for Fast Video Object Segmentation
• Task-adaptive Neural Process for User Cold-Start Recommendation
• Revisiting Mid-Level Patterns for Cross-Domain Few-Shot Recognition
• Adaptive Semiparametric Language Models
• A Hybrid Approach with Optimization and Metric-based Meta-Learner for Few-Shot Learning
• Learning to Control Latent Representations for Few-Shot Learning of Named Entities
• Contextual Memory Trees
• Marrying up Regular Expressions with Neural Networks: A Case Study for Spoken Language Understanding
• Few-shot Learning with Global Relatedness Decoupled-Distillation
• Improving Sample Efficiency with Normalized RBF Kernels
• Local Contrast Learning
• Video SemNet: Memory-Augmented Video Semantic Network
• Graceful Forgetting II. Data as a Process
• CNN with large memory layers
• Learning from Web Data with Self-Organizing Memory Module