PR Product: A Substitute for Inner Product in Neural Networks

Abstract

In this paper, we analyze the inner product of weight vector w and data vector x in neural networks from the perspective of vector orthogonal decomposition and prove that the direction gradient of w decreases with the angle between them close to 0 or π. We propose the Projection and Rejection Product (PR Product) to make the direction gradient of w independent of the angle and consistently larger than the one in standard inner product while keeping the forward propagation identical. As a reliable substitute for standard inner product, the PR Product can be applied into many existing deep learning modules, so we develop the PR Product version of fully connected layer, convolutional layer and LSTM layer. In static image classification, the experiments on CIFAR10 and CIFAR100 datasets demonstrate that the PR Product can robustly enhance the ability of various state-of-the-art classification networks. On the task of image captioning, even without any bells and whistles, our PR Product version of captioning model can compete or outperform the state-of-the-art models on MS COCO dataset. Code has been made available at:https://github.com/wzn0828/PR_Product.

arXiv commentICCV2019 oral

References in corpus (16)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• Wide Residual Networks
• Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
• Training Very Deep Networks
• One weird trick for parallelizing convolutional neural networks
• Improved Image Captioning via Policy Gradient optimization of SPIDEr
• Context-Aware Visual Policy Network for Sequence-Level Image Captioning
• Review Networks for Caption Generation
• A Theory of Local Learning, the Learning Channel, and the Optimality of Backpropagation
• Regularizing RNNs for Caption Generation by Reconstructing The Past with The Present
• Auto-Encoding Scene Graphs for Image Captioning
• MAT: A Multimodal Attentive Translator for Image Captioning