Efficient Representation of Low-Dimensional Manifolds using Deep Networks

Abstract

We consider the ability of deep neural networks to represent data that lies near a low-dimensional manifold in a high-dimensional space. We show that deep networks can efficiently extract the intrinsic, low-dimensional coordinates of such data. We first show that the first two layers of a deep network can exactly embed points lying on a monotonic chain, a special type of piecewise linear manifold, mapping them to a low-dimensional Euclidean space. Remarkably, the network can do this using an almost optimal number of parameters. We also show that this network projects nearby points onto the manifold and then embeds them with little error. We then extend these results to more general manifolds.

References in corpus (7)

• Do Deep Nets Really Need to be Deep?
• On the Number of Linear Regions of Deep Neural Networks
• The Power of Depth for Feedforward Neural Networks
• Provable approximation properties for deep neural networks
• Deep Neural Networks with Random Gaussian Weights: A Universal Classification Strategy?
• On the Expressive Power of Deep Learning: A Tensor Analysis
• Representation Benefits of Deep Feedforward Networks

Cited by in corpus (10)

• Intrinsic dimension of data representations in deep neural networks
• Deep ReLU network approximation of functions on a manifold
• Gradient Dynamics of Shallow Univariate ReLU Networks
• A Neural Scaling Law from the Dimension of the Data Manifold
• Can I trust you more? Model-Agnostic Hierarchical Explanations
• Deep Radial Kernel Networks: Approximating Radially Symmetric Functions with Deep Networks
• Approximate Newton-based statistical inference using only stochastic gradients
• Geometry of Deep Convolutional Networks
• Construction of neural networks for realization of localized deep learning
• PointShuffleNet: Learning Non-Euclidean Features with Homotopy Equivalence and Mutual Information