Branch Geometry and Finite-Radius Sensitivity of Hard-ReLU Training

Xiaoyang Li, Runni Zhou, Xinghao Yan

Outer-learning algorithms use infinitesimal sensitivities to propose finite changes to initialization or training parameters. For hard-ReLU training, the derivative of the finite p…