Exceptional Singularities in Polynomial Neural Networks

Abstract

The singularities of a neurovariety have so far always turned out to lie at subnetworks: functions already computed by a smaller architecture of the same kind. This is what makes them readable as a bias of training towards simpler models, and whether they exhaust the singular locus is left open in the literature. We show that they do not. There are singular points at which every neuron is active, no two neurons are redundant, and the parametrisation is unramified, so that nothing visible at a single parameter signals the singularity. We call these exceptional, and we show that, after quotienting by the group of architectural symmetries, which acts freely on the full parameters, they can arise in only three ways: a rank drop of the differential of the parametrisation, a disconnected fibre, or a failure of properness. A small dimensional example is a fully connected network with two hidden layers of width two and quadratic activations. All the examples we produce are of the second kind, so that what fails at an exceptional singularity is not the simplicity of the model but the possibility of recovering the parameter, up to symmetry, from the function it computes.

arXiv commentTitle changed; substantially revised and extended. The proof of Theorem 35 of v1 is invalid: it deduces that an injective unramified morphism is an immersion, which fails without properness. That theorem should be regarded as an open question; the present version shows that failure of properness is a genuine source of singularities