Energy-Based Models with Applications to Speech and Language Processing

Abstract

Energy-Based Models (EBMs) are an important class of probabilistic models, also known as random fields and undirected graphical models. EBMs are un-normalized and thus radically different from other popular self-normalized probabilistic models such as hidden Markov models (HMMs), autoregressive models, generative adversarial nets (GANs) and variational auto-encoders (VAEs). Over the past years, EBMs have attracted increasing interest not only from the core machine learning community, but also from application domains such as speech, vision, natural language processing (NLP) and so on, due to significant theoretical and algorithmic progress. The sequential nature of speech and language also presents special challenges and needs a different treatment from processing fix-dimensional data (e.g., images). Therefore, the purpose of this monograph is to present a systematic introduction to energy-based models, including both algorithmic progress and applications in speech and language processing. First, the basics of EBMs are introduced, including classic models, recent models parameterized by neural networks, sampling methods, and various learning methods from the classic learning algorithms to the most advanced ones. Then, the application of EBMs in three different scenarios is presented, i.e., for modeling marginal, conditional and joint distributions, respectively. 1) EBMs for sequential data with applications in language modeling, where the main focus is on the marginal distribution of a sequence itself; 2) EBMs for modeling conditional distributions of target sequences given observation sequences, with applications in speech recognition, sequence labeling and text generation; 3) EBMs for modeling joint distributions of both sequences of observations and targets, and their applications in semi-supervised learning and calibrated natural language understanding.

arXiv commentThe version before publisher editing

References in corpus (37)

• Sequence to Sequence Learning with Neural Networks
• A Simple Framework for Contrastive Learning of Visual Representations
• An Introduction to Variational Autoencoders
• BERTScore: Evaluating Text Generation with BERT
• Recurrent Neural Network Regularization
• FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence
• On Calibration of Modern Neural Networks
• Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
• Semi-Supervised Learning with Deep Generative Models
• Temporal Ensembling for Semi-Supervised Learning
• Sequence Transduction with Recurrent Neural Networks
• The Curious Case of Neural Text Degeneration
• Sequence Level Training with Recurrent Neural Networks
• FastSpeech 2: Fast and High-Quality End-to-End Text to Speech
• Stochastic Gradient Hamiltonian Monte Carlo
• A note on the evaluation of generative models
• Good Semi-supervised Learning that Requires a Bad GAN
• A Complete Recipe for Stochastic Gradient MCMC
• A Theory of Generative ConvNet
• BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model
• Deep Directed Generative Models with Energy-Based Probability Estimation
• Learning Approximate Inference Networks for Structured Prediction
• COLD Decoding: Energy-based Constrained Text Generation with Langevin Dynamics
• WenetSpeech: A 10000+ Hours Multi-domain Mandarin Corpus for Speech Recognition
• A Distributional Approach to Controlled Text Generation
• A Review of Learning with Deep Generative Models from Perspective of Graphical Modeling
• Markovian Score Climbing: Variational Inference with KL(p||q)
• Global Normalization for Streaming Speech Recognition in a Modular Framework
• Does Higher Order LSTM Have Better Accuracy for Segmenting and Labeling Sequence Data?
• Backpropagation at the Infinitesimal Inference Limit of Energy-Based Models: Unifying Predictive Coding, Equilibrium Propagation, and Contrastive Hebbian Learning
• Model Interpolation with Trans-dimensional Random Field Language Models for Speech Recognition
• Advancing CTC-CRF Based End-to-End Speech Recognition with Wordpieces and Conformers
• Joint Stochastic Approximation and Its Application to Learning Discrete Latent Variable Models
• Markov Chain Score Ascent: A Unifying Framework of Variational Inference with Markovian Gradients
• Weak Convergence Rates of Population versus Single-Chain Stochastic Approximation MCMC Algorithms
• Transport Score Climbing: Variational Inference Using Forward KL and Adaptive Neural Transport
• Energy-Based Models For Speech Synthesis