End-to-End Offline Goal-Oriented Dialog Policy Learning via Policy Gradient
arXiv:1712.02838
Abstract
Learning a goal-oriented dialog policy is generally performed offline with supervised learning algorithms or online with reinforcement learning (RL). Additionally, as companies accumulate massive quantities of dialog transcripts between customers and trained human agents, encoder-decoder methods have gained popularity as agent utterances can be directly treated as supervision without the need for utterance-level annotations. However, one potential drawback of such approaches is that they myopically generate the next agent utterance without regard for dialog-level considerations. To resolve this concern, this paper describes an offline RL method for learning from unannotated corpora that can optimize a goal-oriented policy at both the utterance and dialog level. We introduce a novel reward function and use both on-policy and off-policy policy gradient to learn a policy offline without requiring online user interaction or an explicit state space definition.
arXiv commentWorkshop on Conversational AI, NIPS 2017, Long Beach, CA, USA
References in corpus (4)
Cited by in corpus (9)
- Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
- MOReL : Model-Based Offline Reinforcement Learning
- Reinforcement Learning for Generative AI: State of the Art, Opportunities and Open Research Challenges
- Efficient (Soft) Q-Learning for Text Generation with Limited Good Data
- Behavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RL
- Document-editing Assistants and Model-based Reinforcement Learning as a Path to Conversational AI
- Interactive Teaching for Conversational AI
- Reducing Conservativeness Oriented Offline Reinforcement Learning
- Turn-level Dialog Evaluation with Dialog-level Weak Signals for Bot-Human Hybrid Customer Service Systems