AutoHOOT: Automatic High-Order Optimization for Tensors
arXiv:2005.04540
Abstract
High-order optimization methods, including Newton's method and its variants as well as alternating minimization methods, dominate the optimization algorithms for tensor decompositions and tensor networks. These tensor methods are used for data analysis and simulation of quantum systems. In this work, we introduce AutoHOOT, the first automatic differentiation (AD) framework targeting at high-order optimization for tensor computations. AutoHOOT takes input tensor computation expressions and generates optimized derivative expressions. In particular, AutoHOOT contains a new explicit Jacobian / Hessian expression generation kernel whose outputs maintain the input tensors' granularity and are easy to optimize. The expressions are then optimized by both the traditional compiler optimization techniques and specific tensor algebra transformations. Experimental results show that AutoHOOT achieves competitive CPU and GPU performance for both tensor decomposition and tensor network applications compared to existing AD software and other tensor computation libraries with manually written kernels. The tensor methods generated by AutoHOOT are also well-parallelizable, and we demonstrate good scalability on a distributed memory supercomputer.
arXiv comment18 pages, 8 figures
References in corpus (11)
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Matrix Product Density Operators: Simulation of finite-T and dissipative systems
- Tensorizing Neural Networks
- Simulating quantum computation by contracting tensor networks
- TensorLy: Tensor Learning in Python
- Low rank factorization of the Coulomb integrals for periodic coupled cluster theory
- TensorNetwork: A Library for Physics and Machine Learning
- Approximate tensor-product preconditioners for very high order discontinuous Galerkin methods
- Tangent: Automatic differentiation using source-code transformation for dynamically typed array programming
- Comparison of Accuracy and Scalability of Gauss-Newton and Alternating Least Squares for CP Decomposition
- A Unified Iteration Space Transformation Framework for Sparse and Dense Tensor Algebra