Re-parameterizing Your Optimizers rather than Architectures

Xiaohan Ding, Honghao Chen, Xiangyu Zhang +3

The well-designed structures in neural networks reflect the prior knowledge incorporated into the models. However, though different models have various priors, we are used to train…