HyperTransfer: Understanding the Equivalence between Base Optimizer and Hyperball

Jinghui Yuan, Hongtao Zhang, Jade Zou +4

Hyperball optimizers constrain parameter norms and update only their directions, establishing a distinct paradigm for neural network optimization. Although this geometry appears fu…