Neural Tangent Kernel Maximum Mean Discrepancy
arXiv:2106.03227
Abstract
We present a novel neural network Maximum Mean Discrepancy (MMD) statistic by identifying a new connection between neural tangent kernel (NTK) and MMD. This connection enables us to develop a computationally efficient and memory-efficient approach to compute the MMD statistic and perform NTK based two-sample tests towards addressing the long-standing challenge of memory and computational complexity of the MMD statistic, which is essential for online implementation to assimilating new samples. Theoretically, such a connection allows us to understand the NTK test statistic properties, such as the Type-I error and testing power for performing the two-sample test, by adapting existing theories for kernel MMD. Numerical experiments on synthetic and real-world datasets validate the theory and demonstrate the effectiveness of the proposed NTK-MMD statistic.
References in corpus (16)
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks
- A Convergence Theory for Deep Learning via Over-Parameterization
- Generative Moment Matching Networks
- Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
- On Lazy Training in Differentiable Programming
- The Principles of Deep Learning Theory
- Graph Neural Tangent Kernel: Fusing Graph Neural Networks with Graph Kernels
- On integral probability metrics, ϕ-divergences and binary classification
- Enhanced Convolutional Neural Tangent Kernels
- When Do Neural Networks Outperform Kernel Methods?
- Fast Two-Sample Testing with Analytic Representations of Probability Measures
- Maximum Mean Discrepancy Gradient Flow
- Classifying high-dimensional Gaussian mixtures: Where kernel methods fail and neural networks succeed
- Limitations of Lazy Training of Two-layers Neural Networks
- Kernel-Based Smoothness Analysis of Residual Networks
- Spectral Analysis of the Neural Tangent Kernel for Deep Residual Networks