A Mechanism for Producing Aligned Latent Spaces with Autoencoders
arXiv:2106.15456
Abstract
Aligned latent spaces, where meaningful semantic shifts in the input space correspond to a translation in the embedding space, play an important role in the success of downstream tasks such as unsupervised clustering and data imputation. In this work, we prove that linear and nonlinear autoencoders produce aligned latent spaces by stretching along the left singular vectors of the data. We fully characterize the amount of stretching in linear autoencoders and provide an initialization scheme to arbitrarily stretch along the top directions using these networks. We also quantify the amount of stretching in nonlinear autoencoders in a simplified setting. We use our theoretical results to align drug signatures across cell types in gene expression space and semantic shifts in word embedding spaces.
References in corpus (10)
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Auto-Encoding Variational Bayes
- Empirical Evaluation of Rectified Activations in Convolutional Network
- Reconciling modern machine learning practice and the bias-variance trade-off
- word2vec Parameter Learning Explained
- A mathematical theory of semantic development in deep neural networks
- From Principal Subspaces to Principal Components with Linear Autoencoders
- Causal Network Models of SARS-CoV-2 Expression and Aging to Identify Candidates for Drug Repurposing
- Learning Word Embeddings from Speech
- Memorization in Overparameterized Autoencoders