On the effect of pooling on the geometry of representations
arXiv:1703.06726
Abstract
In machine learning and neuroscience, certain computational structures and algorithms are known to yield disentangled representations without us understanding why, the most striking examples being perhaps convolutional neural networks and the ventral stream of the visual cortex in humans and primates. As for the latter, it was conjectured that representations may be disentangled by being flattened progressively and at a local scale. An attempt at a formalization of the role of invariance in learning representations was made recently, being referred to as I-theory. In this framework and using the language of differential geometry, we show that pooling over a group of transformations of the input contracts the metric and reduces its curvature, and provide quantitative bounds, in the aim of moving towards a theoretical understanding on how to disentangle representations.
References in corpus (7)
- Understanding Deep Convolutional Networks
- Deep Learning of Representations: Looking Forward
- Unsupervised Learning of Invariant Representations in Hierarchical Architectures
- Transformation Properties of Learned Visual Representations
- Understanding deep features with computer-generated imagery
- Learning Stable Group Invariant Representations with Convolutional Networks
- Local Group Invariant Representations via Orbit Embeddings