VFunc: a Deep Generative Model for Functions

Abstract

We introduce a deep generative model for functions. Our model provides a joint distribution p(f, z) over functions f and latent variables z which lets us efficiently sample from the marginal p(f) and maximize a variational lower bound on the entropy H(f). We can thus maximize objectives of the form E_{f~p(f)}[R(f)] + c*H(f), where R(f) denotes, e.g., a data log-likelihood term or an expected reward. Such objectives encompass Bayesian deep learning in function space, rather than parameter space, and Bayesian deep RL with representations of uncertainty that offer benefits over bootstrapping and parameter noise. In this short paper we describe our model, situate it in the context of prior work, and present proof-of-concept experiments for regression and RL.

arXiv commentTo be presented at the ICML 2018 workshop on Prediction and Generative Modeling in Reinforcement Learning

References in corpus (19)

• Continuous control with deep reinforcement learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
• Weight Uncertainty in Neural Networks
• Deep Bayesian Active Learning with Image Data
• Noisy Networks for Exploration
• MINE: Mutual Information Neural Estimation
• Parameter Space Noise for Exploration
• Learning with Pseudo-Ensembles
• Bridging the Gap Between Value and Policy Based Reinforcement Learning
• Meta-Reinforcement Learning of Structured Exploration Strategies
• Meta Learning Shared Hierarchies
• A unified view of entropy-regularized Markov decision processes
• Diversity is All You Need: Learning Skills without a Reward Function
• Meta-Learning and Universality: Deep Representations and Gradient Descent can Approximate any Learning Algorithm
• Latent Space Policies for Hierarchical Reinforcement Learning
• Implicit Weight Uncertainty in Neural Networks
• A Tutorial on Thompson Sampling
• Bayesian Hypernetworks

Cited by in corpus (7)

• Attentive Neural Processes
• Meta-Learning surrogate models for sequential decision making
• One Solution is Not All You Need: Few-Shot Extrapolation via Structured MaxEnt RL
• Entropy Regularization with Discounted Future State Distribution in Policy Gradient Methods
• Marginalized State Distribution Entropy Regularization in Policy Optimization
• OR-Net: Pointwise Relational Inference for Data Completion under Partial Observation
• Transfer Learning by Modeling a Distribution over Policies