The Loss Surface of XOR Artificial Neural Networks

Abstract

Training an artificial neural network involves an optimization process over the landscape defined by the cost (loss) as a function of the network parameters. We explore these landscapes using optimisation tools developed for potential energy landscapes in molecular science. The number of local minima and transition states (saddle points of index one), as well as the ratio of transition states to minima, grow rapidly with the number of nodes in the network. There is also a strong dependence on the regularisation parameter, with the landscape becoming more convex (fewer minima) as the regularisation term increases. We demonstrate that in our formulation, stationary points for networks with $N_h$ hidden nodes, including the minimal network required to fit the XOR data, are also stationary points for networks with $N_h+1$ hidden nodes when all the weights involving the additional nodes are zero. Hence, smaller networks optimized to train the XOR data are embedded in the landscapes of larger networks. Our results clarify certain aspects of the classification and sensitivity (to perturbations in the input data) of minima and saddle points for this system, and may provide insight into dropout and network compression.

arXiv comment19 pages, 6 figures. Submitted to journal in Oct, 2017

References in corpus (21)

• Global Optimization by Basin-Hopping and the Lowest Energy Structures of Lennard-Jones Clusters Containing up to 110 Atoms
• Spin-Glass Theory for Pedestrians
• A Doubly Nudged Elastic Band Method for Finding Transition States
• Why does deep and cheap learning work so well?
• The double-funnel energy landscape of the 38-atom Lennard-Jones cluster
• Saddle Points and Dynamics of Lennard-Jones Clusters, Solids and Supercooled Liquids
• Thermodynamics and the Global Optimization of Lennard-Jones clusters
• Finding pathways between distant local minima
• Some Further Results for the Stationary Points and Dynamics of Supercooled Liquids
• Exploring the Free Energy Landscape: From Dynamics to Networks and Back
• Complexity of random smooth functions on the high-dimensional sphere
• Perspective: Energy Landscapes for Machine Learning
• Finding All the Stationary Points of a Potential Energy Landscape via Numerical Polynomial Homotopy Continuation Method
• Phase transitions detached from stationary points of the energy landscape
• Topology trivialization and large deviations for the minimum in the simplest random optimization
• Energy landscape analysis of the two-dimensional nearest-neighbor ϕ^4 model
• Stationary point analysis of the one-dimensional lattice Landau gauge fixing functional, aka random phase XY Hamiltonian
• Energy Landscape of the Finite-Size Mean-field 3-Spin Spherical Model
• Potential Energy Landscape of the Two-Dimensional XY Model: Higher-Index Stationary Points
• Energy Landscape of the Finite-Size Mean-field 2-Spin Spherical Model and Topology Trivialization
• Statistics of Stationary Points of Random Finite Polynomial Potentials

Cited by in corpus (4)

• Geometry of energy landscapes and the optimizability of deep neural networks
• Empirical Loss Landscape Analysis of Neural Network Activation Functions
• A Continuous Optimisation Benchmark Suite from Neural Network Regression
• Dissecting a Small Artificial Neural Network