Information flows of diverse autoencoders
arXiv:2102.07402 · doi:10.3390/e23070862
Abstract
The outstanding performance of deep learning in various fields has been a fundamental query, which can be potentially examined using information theory that interprets the learning process as the transmission and compression of information. Information plane analyses of the mutual information between the input-hidden-output layers demonstrated two distinct learning phases of fitting and compression. It is debatable if the compression phase is necessary to generalize the input-output relations extracted from training data. In this study, we investigated this through experiments with various species of autoencoders and evaluated their information processing phase with an accurate kernel-based estimator of mutual information. Given sufficient training data, vanilla autoencoders demonstrated the compression phase, which was amplified after imposing sparsity regularization for hidden activities. However, we found that the compression phase is not universally observed in different species of autoencoders, including variational autoencoders, that have special constraints on network weights or manifold of hidden space. These types of autoencoders exhibited perfect generalization ability for test data without requiring the compression phase. Thus, we conclude that the compression phase is not necessary for generalization in representation learning.
References in corpus (8)
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
- The information bottleneck method
- Opening the Black Box of Deep Neural Networks via Information
- Semantic Autoencoder for Zero-Shot Learning
- On the Information Bottleneck Problems: Models, Connections, Applications and Information Theoretic Views
- Estimating Mixture Entropy with Pairwise Distances
- Information Plane Analysis of Deep Neural Networks via Matrix-Based Renyi's Entropy and Tensor Kernels
- On the Information Plane of Autoencoders