Structure Regularization for Structured Prediction: Theories and Experiments

Abstract

While there are many studies on weight regularization, the study on structure regularization is rare. Many existing systems on structured prediction focus on increasing the level of structural dependencies within the model. However, this trend could have been misdirected, because our study suggests that complex structures are actually harmful to generalization ability in structured prediction. To control structure-based overfitting, we propose a structure regularization framework via \emph{structure decomposition}, which decomposes training samples into mini-samples with simpler structures, deriving a model with better generalization power. We show both theoretically and empirically that structure regularization can effectively control overfitting risk and lead to better accuracy. As a by-product, the proposed method can also substantially accelerate the training speed. The method and the theoretical results can apply to general graphical models with arbitrary structures. Experiments on well-known tasks demonstrate that our method can easily beat the benchmark systems on those highly-competitive tasks, achieving state-of-the-art accuracies yet with substantially faster training speed.

References in corpus (3)

• HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
• Learning with Structured Sparsity
• Slow Learners are Fast

Cited by in corpus (3)

• Finding Function in Form: Compositional Character Models for Open Vocabulary Word Representation
• Complex Structure Leads to Overfitting: A Structure Regularization Decoding Method for Natural Language Processing
• Structure Regularized Neural Network for Entity Relation Classification for Chinese Literature Text