Analytic Manifold Learning: Unifying and Evaluating Representations for Continuous Control
arXiv:2006.08718
Abstract
We address the problem of learning reusable state representations from streaming high-dimensional observations. This is important for areas like Reinforcement Learning (RL), which yields non-stationary data distributions during training. We make two key contributions. First, we propose an evaluation suite that measures alignment between latent and true low-dimensional states. We benchmark several widely used unsupervised learning approaches. This uncovers the strengths and limitations of existing approaches that impose additional constraints/objectives on the latent space. Our second contribution is a unifying mathematical formulation for learning latent relations. We learn analytic relations on source domains, then use these relations to help structure the latent space when learning on target domains. This formulation enables a more general, flexible and principled way of shaping the latent space. It formalizes the notion of learning independent relations, without imposing restrictive simplifying assumptions or requiring domain-specific information. We present mathematical properties, concrete algorithms for implementation and experimental validation of successful learning and transfer of latent relations.
arXiv commentAdded Section 4: "Imposing AML Relations During Transfer"; expanded description of experiments in Section 5: "Evaluating AML and Latent Space Transfer"
References in corpus (9)
- Adam: A Method for Stochastic Optimization
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- DeepMind Control Suite
- State Representation Learning for Control: An Overview
- Multi-Object Representation Learning with Iterative Variational Inference
- Disentangled Sequential Autoencoder
- S-RL Toolbox: Environments, Datasets and Evaluation Metrics for State Representation Learning
- A Non-generative Framework and Convex Relaxations for Unsupervised Learning
- Incremental Slow Feature Analysis: Adaptive and Episodic Learning from High-Dimensional Input Streams