Reinforcement Learning for Generative AI: State of the Art, Opportunities and Open Research Challenges

Abstract

Generative Artificial Intelligence (AI) is one of the most exciting developments in Computer Science of the last decade. At the same time, Reinforcement Learning (RL) has emerged as a very successful paradigm for a variety of machine learning tasks. In this survey, we discuss the state of the art, opportunities and open research questions in applying RL to generative AI. In particular, we will discuss three types of applications, namely, RL as an alternative way for generation without specified objectives; as a way for generating outputs while concurrently maximizing an objective function; and, finally, as a way of embedding desired characteristics, which cannot be easily captured by means of an objective function, into the generative process. We conclude the survey with an in-depth discussion of the opportunities and challenges in this fascinating emerging area.

arXiv commentPublished in JAIR at https://www.jair.org/index.php/jair/article/view/15278

References in corpus (87)

• Auto-Encoding Variational Bayes
• Sequence to Sequence Learning with Neural Networks
• Continuous control with deep reinforcement learning
• Denoising Diffusion Probabilistic Models
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Training language models to follow instructions with human feedback
• Trust Region Policy Optimization
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• BERTScore: Evaluating Text Generation with BERT
• Hierarchical Text-Conditional Image Generation with CLIP Latents
• On the Opportunities and Risks of Foundation Models
• Prioritized Experience Replay
• Soft Actor-Critic Algorithms and Applications
• Dueling Network Architectures for Deep Reinforcement Learning
• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• Asynchronous Methods for Deep Reinforcement Learning
• Deep Unsupervised Learning using Nonequilibrium Thermodynamics
• A Deep Reinforced Model for Abstractive Summarization
• Zero-Shot Text-to-Image Generation
• On the Properties of Neural Machine Translation: Encoder-Decoder Approaches
• The Curious Case of Neural Text Degeneration
• Sequence Level Training with Recurrent Neural Networks
• Gemini: A Family of Highly Capable Multimodal Models
• High-Resolution Image Synthesis with Latent Diffusion Models
• LaMDA: Language Models for Dialog Applications
• Objective-Reinforced Generative Adversarial Networks (ORGAN) for Sequence Generation Models
• MolGAN: An implicit generative model for small molecular graphs
• Recurrent World Models Facilitate Policy Evolution
• Learning Deep Generative Models of Graphs
• Fine-Tuning Language Models from Human Preferences
• Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
• Cooperative Inverse Reinforcement Learning
• Constitutional AI: Harmlessness from AI Feedback
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient
• Reflexion: Language Agents with Verbal Reinforcement Learning
• Graph Convolutional Policy Network for Goal-Directed Molecular Graph Generation
• Long Text Generation via Adversarial Training with Leaked Information
• Why We Need New Evaluation Metrics for NLG
• Dream to Control: Learning Behaviors by Latent Imagination
• Improving alignment of dialogue agents via targeted human judgements
• Scalable agent alignment via reward modeling: a research direction
• ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation
• On the Creativity of Large Language Models
• Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback
• Improving Factuality and Reasoning in Language Models through Multiagent Debate
• A survey on intrinsic motivation in reinforcement learning
• CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning
• Artist Agent: A Reinforcement Learning Approach to Automatic Stroke Generation in Oriental Ink Painting
• Controllable Neural Story Plot Generation via Reward Shaping
• On-line Active Reward Learning for Policy Optimisation in Spoken Dialogue Systems
• Copyright in Generative Deep Learning
• Collective Constitutional AI: Aligning a Language Model with Public Input
• RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback
• Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization
• Fundamental Limitations of Alignment in Large Language Models
• Quark: Controllable Text Generation with Reinforced Unlearning
• Data Generation as Sequential Decision Making
• Learning to Paint With Model-based Deep Reinforcement Learning
• Optimizing Prompts for Text-to-Image Generation
• Graph Constrained Reinforcement Learning for Natural Language Action Spaces
• Aligning Text-to-Image Models using Human Feedback
• Composite Task-Completion Dialogue Policy Learning via Hierarchical Deep Reinforcement Learning
• Design Guidelines for Prompt Engineering Text-to-Image Generative Models
• End-to-End Offline Goal-Oriented Dialog Policy Learning via Policy Gradient
• Learning to Extract Coherent Summary via Deep Reinforcement Learning
• Interactive and Visual Prompt Engineering for Ad-hoc Task Adaptation with Large Language Models
• Toward Diverse Text Generation with Inverse Reinforcement Learning
• Self-collaboration Code Generation via ChatGPT
• Toxicity in ChatGPT: Analyzing Persona-assigned Language Models
• Training Diffusion Models with Reinforcement Learning
• Learning to Sketch with Deep Q Networks and Demonstrated Strokes
• Curiosity in exploring chemical space: Intrinsic rewards for deep molecular reinforcement learning
• Preference Ranking Optimization for Human Alignment
• On the Weaknesses of Reinforcement Learning for Neural Machine Translation
• Offline RL for Natural Language Generation with Implicit Language Q Learning
• Execution-based Code Generation using Deep Reinforcement Learning
• Batch Policy Gradient Methods for Improving Neural Conversation Models
• ColdGANs: Taming Language GANs with Cautious Sampling Strategies
• Generative Cooperative Networks for Natural Language Generation
• Causal Confusion and Reward Misidentification in Preference-Based Reward Learning
• Dialogue Generation: From Imitation Learning to Inverse Reinforcement Learning
• Towards Coherent and Cohesive Long-form Text Generation
• Neural Collage Transfer: Artistic Reconstruction via Material Manipulation
• Rainier: Reinforced Knowledge Introspector for Commonsense Question Answering
• Combining Semantic Guidance and Deep Reinforcement Learning For Generating Human Level Paintings

Cited by in corpus (2)

• Creativity and Machine Learning: A Survey
• Quantum Chemistry Driven Molecular Inverse Design with Data-free Reinforcement Learning