Unsupervised Discovery of Temporal Structure in Noisy Data with Dynamical Components Analysis

Abstract

Linear dimensionality reduction methods are commonly used to extract low-dimensional structure from high-dimensional data. However, popular methods disregard temporal structure, rendering them prone to extracting noise rather than meaningful dynamics when applied to time series data. At the same time, many successful unsupervised learning methods for temporal, sequential and spatial data extract features which are predictive of their surrounding context. Combining these approaches, we introduce Dynamical Components Analysis (DCA), a linear dimensionality reduction method which discovers a subspace of high-dimensional time series data with maximal predictive information, defined as the mutual information between the past and future. We test DCA on synthetic examples and demonstrate its superior ability to extract dynamical structure compared to commonly used linear methods. We also apply DCA to several real-world datasets, showing that the dimensions extracted by DCA are more useful than those extracted by other methods for predicting future states and decoding auxiliary variables. Overall, DCA robustly extracts dynamical structure in noisy, high-dimensional data while retaining the computational efficiency and geometric interpretability of linear dimensionality reduction methods.

arXiv comment22 pages, 10 figures; updated appendix with additional analyses

References in corpus (14)

• Efficient Estimation of Word Representations in Vector Space
• Representation Learning with Contrastive Predictive Coding
• Estimating Mutual Information
• The information bottleneck method
• Character-Aware Neural Language Models
• Linear Dimensionality Reduction: Survey, Insights, and Generalizations
• Least Dependent Component Analysis Based on Mutual Information
• Adaptive, locally-linear models of complex dynamics
• Formal Limitations on the Measurement of Mutual Information
• Nearly Maximally Predictive Features and Their Dimensions
• Dynamical Component Analysis (DyCA): Dimensionality Reduction For High-Dimensional Deterministic Time-Series
• Graph-based Predictable Feature Analysis
• Dynamical Component Analysis (DyCA) and its application on epileptic EEG
• Information Theoretic Co-Training

Cited by in corpus (2)

• Intelligence, physics and information -- the tradeoff between accuracy and simplicity in machine learning
• Representation Learning for Sequence Data with Deep Autoencoding Predictive Components