Statistical-mechanical study of deep Boltzmann machine given weight parameters after training by singular value decomposition

Abstract

Deep learning methods relying on multi-layered networks have been actively studied in a wide range of fields in recent years, and deep Boltzmann machines(DBMs) is one of them. In this study, a model of DBMs with some properites of weight parameters obtained by learning is studied theoretically by a statistical-mechanical approach based on the replica method. The phases characterizing the role of DBMs as a generator and their phase diagram are derived, depending meaningfully on the numbers of layers and units in each layer. It is found, in particular, that the correlation between the weight parameters in the hidden layers plays an essential role and that an increase in the layer number has a negative effect on DBM's performance as a generator when the correlation is smaller than a certain threshold value.

12 pages, 10 figures

References in corpus (5)

• Mean-field message-passing equations in the Hopfield model and its generalizations
• Restricted Boltzmann Machine, recent advances and mean-field theory
• Spectral Dynamics of Learning Restricted Boltzmann Machines
• Annealing and replica-symmetry in Deep Boltzmann Machines
• The solution of the deep Boltzmann machine on the Nishimori line

Cited by in corpus (4)

• Learning a Restricted Boltzmann Machine using biased Monte Carlo sampling
• Hopfield model with planted patterns: a teacher-student self-supervised learning model
• Modeling Structured Data Learning with Restricted Boltzmann Machines in the Teacher-Student Setting
• High-dimensional Asymptotics of VAEs: Threshold of Posterior Collapse and Dataset-Size Dependence of Rate-Distortion Curve