Neural Empirical Bayes
arXiv:1903.02334
Abstract
We unify and and address a set of problems in unsupervised learning with a geometric interpretation of those methods, rooted in the phenomenon. Kernel density is viewed symbolically as where the random variable is smoothed to , and empirical Bayes is the machinery to denoise in a least-squares sense, which we express as . A learning objective is derived by combining these two, symbolically captured by . Crucially, instead of using the original nonparametric estimators, we parametrize with a neural network denoted by ; at optimality, where is the density of . The optimization problem is abstracted as interactions of high-dimensional spheres which emerge due to the concentration of isotropic gaussians. We introduce two algorithmic frameworks based on this machinery: (i) a "walk-jump" sampling scheme that combines Langevin MCMC (walks) and empirical Bayes (jumps), and (ii) a probabilistic framework for , called NEBULA, defined à la Hopfield by the of the learned energy to a set of attractors. We finish the paper by reporting the emergence of very rich "creative memories" as attractors of NEBULA for highly-overlapping spheres.
arXiv comment23 pages, 10 figures
References in corpus (8)
- Adam: A Method for Stochastic Optimization
- Searching for Activation Functions
- On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization
- A review of mean-shift algorithms for clustering
- Deep Energy Estimator Networks
- Efficient and optimal binary Hopfield associative memory storage using minimum probability flow
- On approximating with neural networks
- Associative content-addressable networks with exponentially many robust stable states