Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-control

Abstract

This paper proposes a general method for improving the structure and quality of sequences generated by a recurrent neural network (RNN), while maintaining information originally learned from data, as well as sample diversity. An RNN is first pre-trained on data using maximum likelihood estimation (MLE), and the probability distribution over the next token in the sequence learned by this model is treated as a prior policy. Another RNN is then trained using reinforcement learning (RL) to generate higher-quality outputs that account for domain-specific incentives while retaining proximity to the prior policy of the MLE RNN. To formalize this objective, we derive novel off-policy RL methods for RNNs from KL-control. The effectiveness of the approach is demonstrated on two applications; 1) generating novel musical melodies, and 2) computational molecular generation. For both problems, we show that the proposed method improves the desired properties and structure of the generated sequences, while maintaining information learned from data.

arXiv commentAdd supplementary material

References in corpus (14)

• Adam: A Method for Stochastic Optimization
• Continuous control with deep reinforcement learning
• Playing Atari with Deep Reinforcement Learning
• WaveNet: A Generative Model for Raw Audio
• Prioritized Experience Replay
• Sequence Level Training with Recurrent Neural Networks
• How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation
• Modeling Temporal Dependencies in High-Dimensional Sequences: Application to Polyphonic Music Generation and Transcription
• Deep Reinforcement Learning for Dialogue Generation
• An Actor-Critic Algorithm for Sequence Prediction
• Music transcription modelling and composition using deep learning
• Generating Focussed Molecule Libraries for Drug Discovery with Recurrent Neural Networks
• Taming the Noise in Reinforcement Learning via Soft Updates
• Neural Adaptive Sequential Monte Carlo

Cited by in corpus (9)

• Objective-Reinforced Generative Adversarial Networks (ORGAN) for Sequence Generation Models
• MidiNet: A Convolutional Generative Adversarial Network for Symbolic-domain Music Generation
• Grammar Variational Autoencoder
• Deeply learning molecular structure-property relationships using attention- and gate-augmented graph convolutional network
• Application of generative autoencoder in de novo molecular design
• Explaining Question Answering Models through Text Generation
• Latent Molecular Optimization for Targeted Therapeutic Design
• From Artificial Neural Networks to Deep Learning for Music Generation -- History, Concepts and Trends
• Harmonic Recomposition using Conditional Autoregressive Modeling