Cosine Normalization: Using Cosine Similarity Instead of Dot Product in Neural Networks

Abstract

Traditionally, multi-layer neural networks use dot product between the output vector of previous layer and the incoming weight vector as the input to activation function. The result of dot product is unbounded, thus increases the risk of large variance. Large variance of neuron makes the model sensitive to the change of input distribution, thus results in poor generalization, and aggravates the internal covariate shift which slows down the training. To bound dot product and decrease the variance, we propose to use cosine similarity or centered cosine similarity (Pearson Correlation Coefficient) instead of dot product in neural networks, which we call cosine normalization. We compare cosine normalization with batch, weight and layer normalization in fully-connected neural networks as well as convolutional networks on the data sets of MNIST, 20NEWS GROUP, CIFAR-10/100 and SVHN. Experiments show that cosine normalization achieves better performance than other normalization techniques.

References in corpus (5)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• Deep Residual Learning for Image Recognition
• Normalization Propagation: A Parametric Technique for Removing Internal Covariate Shift in Deep Networks
• Normalizing the Normalizers: Comparing and Extending Network Normalization Schemes

Cited by in corpus (16)

• Flow-Guided Feature Aggregation for Video Object Detection
• Dynamic Few-Shot Visual Learning without Forgetting
• Hyperspherical Prototype Networks
• von Mises-Fisher Mixture Model-based Deep learning: Application to Face Verification
• Active Neural Localization
• Feature Incay for Representation Regularization
• Extended Batch Normalization
• Rethinking Zero-Shot Learning: A Conditional Visual Classification Perspective
• Ring loss: Convex Feature Normalization for Face Recognition
• Low-Shot Learning with Imprinted Weights
• Normalization in Training U-Net for 2D Biomedical Semantic Segmentation
• Scalable Angular Discriminative Deep Metric Learning for Face Recognition
• Generalized Batch Normalization: Towards Accelerating Deep Neural Networks
• What do CNN neurons learn: Visualization & Clustering
• An Internal Covariate Shift Bounding Algorithm for Deep Neural Networks by Unitizing Layers' Outputs
• An Informal Introduction to Multiplet Neural Networks