Kernel Flows: from learning kernels from data into the abyss

Abstract

Learning can be seen as approximating an unknown function by interpolating the training data. Kriging offers a solution to this problem based on the prior specification of a kernel. We explore a numerical approximation approach to kernel selection/construction based on the simple premise that a kernel must be good if the number of interpolation points can be halved without significant loss in accuracy (measured using the intrinsic RKHS norm $\Vert \cdot \Vert$ associated with the kernel). We first test and motivate this idea on a simple problem of recovering the Green's function of an elliptic PDE (with inhomogeneous coefficients) from the sparse observation of one of its solutions. Next we consider the problem of learning non-parametric families of deep kernels of the form $K_1(F_n(x),F_n(x^{\prime }))$ with $F_{n+1}=(I_d+\epsilon G_{n+1})\circ F_n$ and $G_{n+1}\in \mathrm{Span}\{K_1(F_n(x_i),\cdot )\}$. With the proposed approach constructing the kernel becomes equivalent to integrating a stochastic data driven dynamical system, which allows for the training of very deep (bottomless) networks and the exploration of their properties. These networks learn by constructing flow maps in the kernel and input spaces via incremental data-dependent deformations/perturbations (appearing as the cooperative counterpart of adversarial examples) and, at profound depths, they (1) can achieve accurate classification from only one data point per class (2) appear to learn archetypes of each class (3) expand distances between points that are in different classes and contract distances between points in the same class. For kernels parameterized by the weights of Convolutional Neural Networks, minimizing approximation errors incurred by halving random subsets of interpolation points, appears to outperform training (the same CNN architecture) with relative entropy and dropout.

arXiv comment42 pages, 31 figures. See https://www.youtube.com/watch?v=h9wB8FVH7YM&list=PLdWd7x7FVuLphAODzEvj2KRNws7z7Sv87 for animations of the flows. See http://users.cms.caltech.edu/~owhadi/index_htm_files/kf-static.pdf for slides. See http://users.cms.caltech.edu/~owhadi/KF/ for high resolution videos of the flows in the setting of the Swiss roll cheesecake classification problem

References in corpus (11)

• Adam: A Method for Stochastic Optimization
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Understanding deep learning requires rethinking generalization
• Artificial Intelligence and Statistics
• Polyharmonic homogenization, rough polyharmonic splines and sparse super-localization
• Non-intrusive and structure preserving multiscale integration of stiff ODEs, SDEs and Hamiltonian systems with hidden slow dynamics via flow averaging
• On the Brittleness of Bayesian Inference
• Brittleness of Bayesian Inference Under Finite Information in a Continuous World
• Universal Scalable Robust Solvers from Computational Information Games and fast eigenspace adapted Multiresolution Analysis
• Learning with Hierarchical Gaussian Kernels
• How Deep Are Deep Gaussian Processes?

Cited by in corpus (25)

• Learning dynamical systems from data: a simple cross-validation perspective
• Operator-theoretic framework for forecasting nonlinear time series with kernel analog techniques
• Kernel Learning for Robust Dynamic Mode Decomposition: Linear and Nonlinear Disambiguation Optimization (LANDO)
• Kernel-based approximation of the Koopman generator and Schrödinger operator
• Reproducing kernel Hilbert space compactification of unitary evolution groups
• One-Shot Learning of Stochastic Differential Equations with Data Adapted Kernels
• Learning "best" kernels from data in Gaussian process regression. With application to aerodynamics
• Deep regularization and direct training of the inner layers of Neural Networks with Kernel Flows
• Solving and Learning Nonlinear PDEs with Gaussian Processes
• Numerical methods for Mean field Games based on Gaussian Processes and Fourier Features
• Symmetric and antisymmetric kernels for machine learning problems in quantum physics and chemistry
• Learning dynamical systems from data: A simple cross-validation perspective, part III: Irregularly-Sampled Time Series
• Gaussian Process Regression for Maximum Entropy Distribution
• Hyperparameter Optimization for Randomized Algorithms: A Case Study on Random Features
• Kernel Methods for the Approximation of the Eigenfunctions of the Koopman Operator
• Variational Inference for Nonlinear Inverse Problems via Neural Net Kernels: Comparison to Bayesian Neural Networks, Application to Topology Optimization
• ResNet After All? Neural ODEs and Their Numerical Solution
• Posterior Consistency of Semi-Supervised Regression on Graphs
• Multiclass classification utilising an estimated algorithmic probability prior
• Kernel-based parameter estimation of dynamical systems with unknown observation functions
• Consistency of Empirical Bayes And Kernel Flow For Hierarchical Parameter Estimation
• Analysis of Structured Deep Kernel Networks
• Deep Learning with Kernel Flow Regularization for Time Series Forecasting
• Data-driven geophysical forecasting: Simple, low-cost, and accurate baselines with kernel methods
• Generalized Rough Polyharmonic Splines for Multiscale PDEs with Rough Coefficients