Spectral Gradient Orthogonalization Improves Differentially Private Training at Scale

Sabari Shanmugam, Nick Barnes, Kerry Taylor

Differentially private training adds isotropic Gaussian noise to clipped gradients, corrupting every singular direction equally. In vision models, where spatial correlation concent…