LogAvgExp Provides a Principled and Performant Global Pooling Operator

Abstract

We seek to improve the pooling operation in neural networks, by applying a more theoretically justified operator. We demonstrate that LogSumExp provides a natural OR operator for logits. When one corrects for the number of elements inside the pooling operator, this becomes $\text{LogAvgExp}:=\log (\text{mean}(\exp (x)))$. By introducing a single temperature parameter, LogAvgExp smoothly transitions from the max of its operands to the mean (found at the limiting cases $t\to 0^{+}$ and $t\to +\infty$). We experimentally tested LogAvgExp, both with and without a learnable temperature parameter, in a variety of deep neural network architectures for computer vision.

References in corpus (12)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Deep Residual Learning for Image Recognition
• Wide Residual Networks
• Going Deeper with Convolutions
• AutoAugment: Learning Augmentation Policies from Data
• On the Variance of the Adaptive Learning Rate and Beyond
• Revisiting Multiple Instance Neural Networks
• Aggregated Residual Transformations for Deep Neural Networks
• Fractional Max-Pooling
• Bag of Tricks for Image Classification with Convolutional Neural Networks
• Deep Pyramidal Residual Networks