Approximation Theory of Tree Tensor Networks: Tensorized Univariate Functions -- Part I

Abstract

We study the approximation of functions by tensor networks (TNs). We show that Lebesgue $L^p$-spaces in one dimension can be identified with tensor product spaces of arbitrary order through tensorization. We use this tensor product structure to define subsets of $L^p$ of rank-structured functions of finite representation complexity. These subsets are then used to define different approximation classes of tensor networks, associated with different measures of complexity. These approximation classes are shown to be quasi-normed linear spaces. We study some elementary properties and relationships of said spaces. In part II of this work, we will show that classical smoothness (Besov) spaces are continuously embedded into these approximation classes. We will also show that functions in these approximation classes do not possess any Besov smoothness, unless one restricts the depth of the tensor networks. The results of this work are both an analysis of the approximation spaces of TNs and a study of the expressivity of a particular type of neural networks (NN) -- namely feed-forward sum-product networks with sparse architecture. The input variables of this network result from the tensorization step, interpreted as a particular featuring step which can also be implemented with a neural network with a specific architecture. We point out interesting parallels to recent results on the expressivity of rectified linear unit (ReLU) networks -- currently one of the most popular type of NNs.

arXiv commentFor part II see arXiv:2007.00128, for part III see arXiv:2101.11932

References in corpus (16)

• Deep Learning in Neural Networks: An Overview
• Solving the Quantum Many-Body Problem with Artificial Neural Networks
• A Practical Introduction to Tensor Networks: Matrix Product States and Projected Entangled Pair States
• An Area Law for One Dimensional Quantum Systems
• Tensor networks for complex quantum systems
• Low-Rank Tensor Networks for Dimensionality Reduction and Large-Scale Optimization Problems: Perspectives and Challenges PART 1
• Tensor Networks for Dimensionality Reduction and Large-Scale Optimizations. Part 2 Applications and Future Perspectives
• Quantum Entanglement in Deep Learning Architectures
• Adaptive Near-Optimal Rank Tensor Approximation for High-Dimensional Operator Equations
• On the Dirac-Frenkel Variational Principle on Tensor Banach Spaces
• Approximating local observables on projected entangled pair states
• Learning with tree tensor networks: complexity estimates and model selection
• Approximation of Smoothness Classes by Deep Rectifier Networks
• Higher-order principal component analysis for the approximation of tensors in tree-based low-rank formats
• Hackbusch Conjecture on tensor formats
• Tensor-based multiscale method for diffusion problems in quasi-periodic heterogeneous media

Cited by in corpus (2)

• Learning with tree tensor networks: complexity estimates and model selection
• A Quantum-Inspired Algorithm for Wave Simulation Using Tensor Networks