Learning Approximate Inference Networks for Structured Prediction

Abstract

Structured prediction energy networks (SPENs; Belanger & McCallum 2016) use neural network architectures to define energy functions that can capture arbitrary dependencies among parts of structured outputs. Prior work used gradient descent for inference, relaxing the structured output to a set of continuous variables and then optimizing the energy with respect to them. We replace this use of gradient descent with a neural network trained to approximate structured argmax inference. This "inference network" outputs continuous values that we treat as the output structure. We develop large-margin training criteria for joint training of the structured energy function and inference network. On multi-label classification we report speed-ups of 10-60x compared to (Belanger et al, 2017) while also improving accuracy. For sequence labeling with simple structured energies, our approach performs comparably to exact inference while being much faster at test time. We then demonstrate improved accuracy by augmenting the energy with a "label language model" that scores entire output label sequences, showing it can improve handling of long-distance dependencies in part-of-speech tagging. Finally, we show how inference networks can replace dynamic programming for test-time inference in conditional random fields, suggestive for their general use for fast inference in structured settings.

arXiv commentaccepted by ICLR2018

References in corpus (6)

• Adam: A Method for Stochastic Optimization
• Distilling the Knowledge in a Neural Network
• Improved Techniques for Training GANs
• Input Convex Neural Networks
• Sequence-Level Knowledge Distillation
• Do Deep Convolutional Nets Really Need to be Deep and Convolutional?

Cited by in corpus (14)

• Hierarchically-Refined Label Attention Network for Sequence Labeling
• Adversarial Contrastive Estimation
• Gaussian Mixture Variational Autoencoder with Contrastive Learning for Multi-Label Classification
• Flow-based Spatio-Temporal Structured Prediction of Motion Dynamics
• Melding the Data-Decisions Pipeline: Decision-Focused Learning for Combinatorial Optimization
• GENN: Predicting Correlated Drug-drug Interactions with Graph Energy Neural Networks
• Structured Output Learning with Conditional Generative Flows
• Amortized Bethe Free Energy Minimization for Learning MRFs
• Neural Message Passing for Multi-Label Classification
• Energy-Based Models with Applications to Speech and Language Processing
• Improving Joint Training of Inference Networks and Structured Prediction Energy Networks
• Adversarially-learned Inference via an Ensemble of Discrete Undirected Graphical Models
• Semantic Role Labeling with Iterative Structure Refinement
• AIN: Fast and Accurate Sequence Labeling with Approximate Inference Network