Uncertainty Estimates for Efficient Neural Network-based Dialogue Policy Optimisation

Abstract

In statistical dialogue management, the dialogue manager learns a policy that maps a belief state to an action for the system to perform. Efficient exploration is key to successful policy optimisation. Current deep reinforcement learning methods are very promising but rely on epsilon-greedy exploration, thus subjecting the user to a random choice of action during learning. Alternative approaches such as Gaussian Process SARSA (GPSARSA) estimate uncertainties and are sample efficient, leading to better user experience, but on the expense of a greater computational complexity. This paper examines approaches to extract uncertainty estimates from deep Q-networks (DQN) in the context of dialogue management. We perform an extensive benchmark of deep Bayesian methods to extract uncertainty estimates, namely Bayes-By-Backprop, dropout, its concrete variation, bootstrapped ensemble and alpha-divergences, combining it with DQN algorithm.

Accepted at the Bayesian Deep Learning Workshop, 31st Conference on Neural Information Processing Systems (NIPS 2017)

Cited by in corpus (2)

• An Improved Approach of Intention Discovery with Machine Learning for POMDP-based Dialogue Management
• "None of the Above":Measure Uncertainty in Dialog Response Retrieval