On Characterizing the Capacity of Neural Networks using Algebraic Topology

Abstract

The learnability of different neural architectures can be characterized directly by computable measures of data complexity. In this paper, we reframe the problem of architecture selection as understanding how data determines the most expressive and generalizable architectures suited to that data, beyond inductive bias. After suggesting algebraic topology as a measure for data complexity, we show that the power of a network to express the topological complexity of a dataset in its decision region is a strictly limiting factor in its ability to generalize. We then provide the first empirical characterization of the topological capacity of neural networks. Our empirical analysis shows that at every level of dataset complexity, neural networks exhibit topological phase transitions. This observation allowed us to connect existing theory to empirically driven conjectures on the choice of architectures for fully-connected neural networks.

13 pages, 11 figures

References in corpus (4)

• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• OpenML: networked science in machine learning
• PathNet: Evolution Channels Gradient Descent in Super Neural Networks
• Don't Decay the Learning Rate, Increase the Batch Size

Cited by in corpus (28)

• Neural Persistence: A Complexity Measure for Deep Neural Networks Using Algebraic Topology
• A Topology Layer for Machine Learning
• A Short Survey of Topological Data Analysis in Time Series and Systems Analysis
• Path homologies of deep feedforward networks
• Deeply Learning Derivatives
• Activation Landscapes as a Topological Summary of Neural Network Performance
• Time Series Featurization via Topological Data Analysis
• Interpreting Deep Learning: The Machine Learning Rorschach Test?
• PLLay: Efficient Topological Layer based on Persistence Landscapes
• RipsNet: a general architecture for fast and robust estimation of the persistent homology of point clouds
• Topology of deep neural networks
• Fuzzy c-Means Clustering for Persistence Diagrams
• Topological Data Analysis in Text Classification: Extracting Features with Additive Information
• Analyzing Hypersensitive AI: Instability in Corporate-Scale Machine Learning
• ANS: Adaptive Network Scaling for Deep Rectifier Reinforcement Learning Models
• Capacity allocation analysis of neural networks: A tool for principled architecture design
• Capacity allocation through neural network layers
• Can VAEs Generate Novel Examples?
• Finding the Homology of Decision Boundaries with Active Learning
• Topological Uncertainty: Monitoring trained neural networks through persistence of activation graphs
• Some Geometrical and Topological Properties of DNNs' Decision Boundaries
• Homology Localization Through the Looking-Glass of Parameterized Complexity Theory
• Abstraction, Reasoning and Deep Learning: A Study of the "Look and Say" Sequence
• Dive into Layers: Neural Network Capacity Bounding using Algebraic Geometry
• Self-Organizing Map assisted Deep Autoencoding Gaussian Mixture Model for Intrusion Detection
• Topologically Densified Distributions
• An empirical study of the relation between network architecture and complexity
• Characterizing and Measuring the Similarity of Neural Networks with Persistent Homology