Learning Energy-Based Approximate Inference Networks for Structured Applications in NLP

Abstract

Structured prediction in natural language processing (NLP) has a long history. The complex models of structured application come at the difficulty of learning and inference. These difficulties lead researchers to focus more on models with simple structure components (e.g., local classifier). Deep representation learning has become increasingly popular in recent years. The structure components of their method, on the other hand, are usually relatively simple. We concentrate on complex structured models in this dissertation. We provide a learning framework for complicated structured models as well as an inference method with a better speed/accuracy/search error trade-off. The dissertation begins with a general introduction to energy-based models. In NLP and other applications, an energy function is comparable to the concept of a scoring function. In this dissertation, we discuss the concept of the energy function and structured models with different energy functions. Then, we propose a method in which we train a neural network to do argmax inference under a structured energy function, referring to the trained networks as "inference networks" or "energy-based inference networks". We then develop ways of jointly learning energy functions and inference networks using an adversarial learning framework. Despite the inference and learning difficulties of energy-based models, we present approaches in this thesis that enable energy-based models more easily to be applied in structured NLP applications.

Ph.D. Thesis

References in corpus (14)

• Distilling the Knowledge in a Neural Network
• Improving neural networks by preventing co-adaptation of feature detectors
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Natural Language Processing (almost) from Scratch
• Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials
• Loopy Belief Propagation for Approximate Inference: An Empirical Study
• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
• Energy-based Generative Adversarial Network
• Regularizing Neural Networks by Penalizing Confident Output Distributions
• Actor-Critic Sequence Training for Image Captioning
• Aligned Cross Entropy for Non-Autoregressive Machine Translation
• Notes on Noise Contrastive Estimation and Negative Sampling
• Minimizing the Bag-of-Ngrams Difference for Non-Autoregressive Neural Machine Translation
• Imitation Learning for Non-Autoregressive Neural Machine Translation