Understanding Gradient Orthogonalization for Deep Learning via Non-Euclidean Trust-Region Optimization

Dmitry Kovalev

Optimization with matrix gradient orthogonalization has recently demonstrated impressive results in the training of deep neural networks (Jordan et al., 2024; Liu et al., 2025). In…