CNN with large memory layers

Abstract

This work is centred around the recently proposed product key memory structure \cite{large_memory}, implemented for a number of computer vision applications. The memory structure can be regarded as a simple computation primitive suitable to be augmented to nearly all neural network architectures. The memory block allows implementing sparse access to memory with square root complexity scaling with respect to the memory capacity. The latter scaling is possible due to the incorporation of Cartesian product space decomposition of the key space for the nearest neighbour search. We have tested the memory layer on the classification, image reconstruction and relocalization problems and found that for some of those, the memory layers can provide significant speed/accuracy improvement with the high utilization of the key-value elements, while others require more careful fine-tuning and suffer from dying keys. To tackle the later problem we have introduced a simple technique of memory re-initialization which helps us to eliminate unused key-value pairs from the memory and engage them in training again. We have conducted various experiments and got improvements in speed and accuracy for classification and PoseNet relocalization models. We showed that the re-initialization has a huge impact on a toy example of randomly labeled data and observed some gains in performance on the image classification task. We have also demonstrated the generalization property perseverance of the large memory layers on the relocalization problem, while observing the spatial correlations between the images and the selected memory cells.

arXiv commentMaster's dissertation paper

References in corpus (27)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Distilling the Knowledge in a Neural Network
• Sequence to Sequence Learning with Neural Networks
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Language Models are Few-Shot Learners
• Neural Discrete Representation Learning
• Understanding deep learning requires rethinking generalization
• End-To-End Memory Networks
• NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis
• $f$-divergence Inequalities
• Learning to Remember Rare Events
• Stand-Alone Self-Attention in Vision Models
• Practical and Optimal LSH for Angular Distance
• Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks
• Conditional Computation in Neural Networks for faster models
• Global Sparse Momentum SGD for Pruning Very Deep Neural Networks
• Neural Episodic Control
• Audio-driven Talking Face Video Generation with Learning-based Personalized Head Pose
• Large Memory Layers with Product Keys
• Variance Reduction for Reinforcement Learning in Input-Driven Environments
• Small ReLU networks are powerful memorizers: a tight analysis of memorization capacity
• ImageNet Training in Minutes
• Exponentially Increasing the Capacity-to-Computation Ratio for Conditional Computation in Deep Learning
• Memory-Based Graph Networks
• Understanding Multi-Head Attention in Abstractive Summarization