GENNI: Visualising the Geometry of Equivalences for Neural Network Identifiability

Abstract

We propose an efficient algorithm to visualise symmetries in neural networks. Typically, models are defined with respect to a parameter space, where non-equal parameters can produce the same input-output map. Our proposed method, GENNI, allows us to efficiently identify parameters that are functionally equivalent and then visualise the subspace of the resulting equivalence class. By doing so, we are now able to better explore questions surrounding identifiability, with applications to optimisation and generalizability, for commonly used or newly developed neural network architectures.

References in corpus (6)

• UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction
• Essentially No Barriers in Neural Network Energy Landscape
• Empirical Analysis of the Hessian of Over-Parametrized Neural Networks
• Weight-space symmetry in deep networks gives rise to permutation saddles, connected by equal-loss valleys across the loss landscape
• $\mathcal{G}$-SGD: Optimizing ReLU Neural Networks in its Positively Scale-Invariant Space
• A Foliated View of Transfer Learning

Cited by in corpus (1)

• Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariances