Representation Learning for Sequence Data with Deep Autoencoding Predictive Components

Abstract

We propose Deep Autoencoding Predictive Components (DAPC) -- a self-supervised representation learning method for sequence data, based on the intuition that useful representations of sequence data should exhibit a simple structure in the latent space. We encourage this latent structure by maximizing an estimate of predictive information of latent feature sequences, which is the mutual information between past and future windows at each time step. In contrast to the mutual information lower bound commonly used by contrastive learning, the estimate of predictive information we adopt is exact under a Gaussian assumption. Additionally, it can be computed without negative sampling. To reduce the degeneracy of the latent space extracted by powerful encoders and keep useful information from the inputs, we regularize predictive information learning with a challenging masked reconstruction loss. We demonstrate that our method recovers the latent space of noisy dynamical systems, extracts predictive features for forecasting tasks, and improves automatic speech recognition when used to pretrain the encoder on large amounts of unlabeled data.

References in corpus (24)

• Auto-Encoding Variational Bayes
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• A Simple Framework for Contrastive Learning of Visual Representations
• Estimating Mutual Information
• SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
• Bootstrap your own latent: A new approach to self-supervised Learning
• Language Models are Few-Shot Learners
• Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
• Attention-Based Models for Speech Recognition
• Learning deep representations by mutual information estimation and maximization
• A Comparative Study on Transformer vs RNN in Speech Applications
• A Recurrent Latent Variable Model for Sequential Data
• Contrastive Multiview Coding
• TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech
• On Mutual Information Maximization for Representation Learning
• Deep Contextualized Acoustic Representations For Semi-Supervised Speech Recognition
• Improving Transformer-based Speech Recognition Using Unsupervised Pre-training
• Effectiveness of self-supervised pre-training for speech recognition
• Amortized Inference Regularization
• Wasserstein Dependency Measure for Representation Learning
• Learning Robust and Multilingual Speech Representations
• Speech-XLNet: Unsupervised Acoustic Model Pretraining For Self-Attention Networks
• Unsupervised Discovery of Temporal Structure in Noisy Data with Dynamical Components Analysis
• Masked Pre-trained Encoder base on Joint CTC-Transformer