Scale redundancy and soft gauge fixing in positively homogeneous neural networks

Rodrigo Carmo Terin

Neural networks with positively homogeneous activations exhibit an exact continuous reparametrization symmetry: neuron-wise rescalings generate parameter-space orbits along which t…