Walking the Tightrope: An Investigation of the Convolutional Autoencoder Bottleneck

Abstract

In this paper, we present an in-depth investigation of the convolutional autoencoder (CAE) bottleneck. Autoencoders (AE), and especially their convolutional variants, play a vital role in the current deep learning toolbox. Researchers and practitioners employ CAEs for a variety of tasks, ranging from outlier detection and compression to transfer and representation learning. Despite their widespread adoption, we have limited insight into how the bottleneck shape impacts the emergent properties of the CAE. We demonstrate that increased height and width of the bottleneck drastically improves generalization, which in turn leads to better performance of the latent codes in downstream transfer learning tasks. The number of channels in the bottleneck, on the other hand, is secondary in importance. Furthermore, we show empirically that, contrary to popular belief, CAEs do not learn to copy their input, even when the bottleneck has the same number of neurons as there are pixels in the input. Copying does not occur, despite training the CAE for 1,000 epochs on a tiny ($\approx$ 600 images) dataset. We believe that the findings in this paper are directly applicable and will lead to improvements in models that rely on CAEs.

arXiv commentcode available at https://github.com/IljaManakov/WalkingTheTightrope

References in corpus (11)

• Instance Normalization: The Missing Ingredient for Fast Stylization
• A guide to convolution arithmetic for deep learning
• A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decay
• Deep Autoencoding Models for Unsupervised Anomaly Segmentation in Brain MR Images
• Image Restoration Using Convolutional Auto-encoders with Symmetric Skip Connections
• Provable Bounds for Learning Some Deep Representations
• SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability
• Qualitatively characterizing neural network optimization problems
• Deforming Autoencoders: Unsupervised Disentangling of Shape and Appearance
• Why Regularized Auto-Encoders learn Sparse Representation?
• Autoencoders Learn Generative Linear Models