Double Preconditioning (DoPr): Optimization for Test-Time Performance, not Validation Loss

Thomas T. Zhang, Alok Shah, Yifei Zhang +3

Many modern applications of deep learning involve training a neural network via a one-step prediction loss (e.g., $L^2$ regression, cross-entropy), but deploy the network by rollin…