Optimal regularizations for data generation with probabilistic graphical models

Abstract

Understanding the role of regularization is a central question in Statistical Inference. Empirically, well-chosen regularization schemes often dramatically improve the quality of the inferred models by avoiding overfitting of the training data. We consider here the particular case of L 2 and L 1 regularizations in the Maximum A Posteriori (MAP) inference of generative pairwise graphical models. Based on analytical calculations on Gaussian multivariate distributions and numerical experiments on Gaussian and Potts models we study the likelihoods of the training, test, and 'generated data' (with the inferred models) sets as functions of the regularization strengths. We show in particular that, at its maximum, the test likelihood and the 'generated' likelihood, which quantifies the quality of the generated samples, have remarkably close values. The optimal value for the regularization strength is found to be approximately equal to the inverse sum of the squared couplings incoming on sites on the underlying network of interactions. Our results seem largely independent of the structure of the true underlying interactions that generated the data, of the regularization scheme considered, and are valid when small fluctuations of the posterior distribution around the MAP estimator are taken into account. Connections with empirical works on protein models learned from homologous sequences are discussed.

References in corpus (17)

• Scikit-learn: Machine Learning in Python
• SciPy 1.0--Fundamental Algorithms for Scientific Computing in Python
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Recurrent Neural Network Regularization
• Quantification of the effect of mutations using a global probability model of natural sequence variation
• Improved contact prediction in proteins: Using pseudolikelihoods to infer Potts models
• High-dimensional Ising model selection using ${\ell }_1$-regularized logistic regression
• Inverse statistical problems: from the inverse Ising problem to data science
• Inverse Statistical Physics of Protein Sequences: A Key Issues Review
• Fast pseudolikelihood maximization for direct-coupling analysis of protein structure from many homologous amino-acid sequences
• Adaptive Cluster Expansion for Inferring Boltzmann Machines with Noisy Data
• Generalisation error in learning with random features and the hidden manifold model
• Surprises in High-Dimensional Ridgeless Least Squares Interpolation
• Influence of Multiple Sequence Alignment Depth on Potts Statistical Models of Protein Covariation
• Sparse generative modeling via parameter-reduction of Boltzmann machines: application to protein-sequence families
• Large Pseudo-Counts and $L_2$-Norm Penalties Are Necessary for the Mean-Field Inference of Ising and Potts Models
• Inference of compressed Potts graphical models