FC2T2: The Fast Continuous Convolutional Taylor Transform with Applications in Vision and Graphics

Abstract

Series expansions have been a cornerstone of applied mathematics and engineering for centuries. In this paper, we revisit the Taylor series expansion from a modern Machine Learning perspective. Specifically, we introduce the Fast Continuous Convolutional Taylor Transform (FC2T2), a variant of the Fast Multipole Method (FMM), that allows for the efficient approximation of low dimensional convolutional operators in continuous space. We build upon the FMM which is an approximate algorithm that reduces the computational complexity of N-body problems from O(NM) to O(N+M) and finds application in e.g. particle simulations. As an intermediary step, the FMM produces a series expansion for every cell on a grid and we introduce algorithms that act directly upon this representation. These algorithms analytically but approximately compute the quantities required for the forward and backward pass of the backpropagation algorithm and can therefore be employed as (implicit) layers in Neural Networks. Specifically, we introduce a root-implicit layer that outputs surface normals and object distances as well as an integral-implicit layer that outputs a rendering of a radiance field given a 3D pose. In the context of Machine Learning, $N$ and $M$ can be understood as the number of model parameters and model evaluations respectively which entails that, for applications that require repeated function evaluations which are prevalent in Computer Vision and Graphics, unlike regular Neural Networks, the techniques introduce in this paper scale gracefully with parameters. For some applications, this results in a 200x reduction in FLOPs compared to state-of-the-art approaches at a reasonable or non-existent loss in accuracy.

References in corpus (12)

• Geometric deep learning: going beyond Euclidean data
• Fourier Neural Operator for Parametric Partial Differential Equations
• Neural Ordinary Differential Equations
• Speeding up Convolutional Neural Networks with Low Rank Expansions
• Differentiable Surface Splatting for Point-based Geometry Processing
• Hypernetwork Knowledge Graph Embeddings
• Multipole Graph Neural Operator for Parametric Partial Differential Equations
• Learning Efficient Point Cloud Generation for Dense 3D Object Reconstruction
• On the Spectral Bias of Neural Networks
• Differentiable Rendering: A Survey
• Dual-Tree Fast Gauss Transforms
• Unsupervised Learning of Shape and Pose with Differentiable Point Clouds