Improving Covariance Conditioning of the SVD Meta-layer by Orthogonality

Yue Song, Nicu Sebe, Wei Wang

Inserting an SVD meta-layer into neural networks is prone to make the covariance ill-conditioned, which could harm the model in the training stability and generalization abilities.…