Failure of entropic selection for diagonal matrix completion in deep linear networks

Abstract

We study the free-energy gradient flow associated with the depth-$N$ deep linear network on the space of invertible real $d\times d$ matrices. The loss observes only the diagonal, \[ E_d(W)=\frac12\sum_{i=1}^d(W_{ii}-1)^2, \] and the regularizer is the Boltzmann entropy of the balanced factorization fiber computed by Menon and Yu. This is a natural higher-dimensional version of a matrix-completion problem posed by Menon as a test of whether entropy selects among a noncompact family of minimizers. For diagonal completion, every width $d\ge 2$, depth $N>2$, and inverse temperature $\beta >0$, we prove that the free energy is unbounded below and has exactly $2^d$ full-rank critical points, one in each diagonal sign chamber. Every critical point is diagonal and hyperbolic. Its unstable dimension is $\left(\genfrac{}{}{0ex}{}{d}{2}\right)$ and its stable dimension is $d(d+1)/2$. We also prove that a full-rank trajectory cannot converge to a finite rank-deficient matrix. Consequently, almost every full-rank initial condition has an unbounded forward orbit. Thus finite-temperature fiber entropy does not provide an equilibrium selection principle for diagonal completion.

arXiv commentComments welcome