Learning Ordered Representations with Nested Dropout

Abstract

In this paper, we study ordered representations of data in which different dimensions have different degrees of importance. To learn these representations we introduce nested dropout, a procedure for stochastically removing coherent nested sets of hidden units in a neural network. We first present a sequence of theoretical results in the simple case of a semi-linear autoencoder. We rigorously show that the application of nested dropout enforces identifiability of the units, which leads to an exact equivalence with PCA. We then extend the algorithm to deep models and demonstrate the relevance of ordered representations to a number of applications. Specifically, we use the ordered property of the learned codes to construct hash-based data structures that permit very fast retrieval, achieving retrieval in time logarithmic in the database size and independent of the dimensionality of the representation. This allows codes that are hundreds of times longer than currently feasible for retrieval. We therefore avoid the diminished quality associated with short codes, while still performing retrieval that is competitive in speed with existing methods. We also show that ordered representations are a promising way to learn adaptive compression for efficient online data reconstruction.

11 pages, 5 figures. Submitted for publication

References in corpus (1)

• Improving neural networks by preventing co-adaptation of feature detectors

Cited by in corpus (23)

• Compressing Neural Networks with the Hashing Trick
• Learning with Pseudo-Ensembles
• Spectral Representations for Convolutional Neural Networks
• Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networks
• Learning Sparse Networks Using Targeted Dropout
• FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropout
• Earliness-Aware Deep Convolutional Networks for Early Time Series Classification
• Learning unbiased features
• Regularized linear autoencoders recover the principal components, eventually
• Variable-rate discrete representation learning
• Disentangling Style and Content in Anime Illustrations
• Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search
• Boosting Co-teaching with Compression Regularization for Label Noise
• Efficient batchwise dropout training using submatrices
• Anytime Sampling for Autoregressive Models via Ordered Autoencoding
• Anime Style Space Exploration Using Metric Learning and Generative Adversarial Networks
• Airfoil Design Parameterization and Optimization using Bézier Generative Adversarial Networks
• Progressive Neural Image Compression with Nested Quantization and Latent Ordering
• Hyperparameter Transfer Learning with Adaptive Complexity
• Improving compute efficacy frontiers with SliceOut
• Dropout with Tabu Strategy for Regularizing Deep Neural Networks
• Ordering Dimensions with Nested Dropout Normalizing Flows
• Rank Ordered Autoencoders