Learning Calibratable Policies using Programmatic Style-Consistency

Abstract

We study the problem of controllable generation of long-term sequential behaviors, where the goal is to calibrate to multiple behavior styles simultaneously. In contrast to the well-studied areas of controllable generation of images, text, and speech, there are two questions that pose significant challenges when generating long-term behaviors: how should we specify the factors of variation to control, and how can we ensure that the generated behavior faithfully demonstrates combinatorially many styles? We leverage programmatic labeling functions to specify controllable styles, and derive a formal notion of style-consistency as a learning objective, which can then be solved using conventional policy learning approaches. We evaluate our framework using demonstrations from professional basketball players and agents in the MuJoCo physics environment, and show that existing approaches that do not explicitly enforce style-consistency fail to generate diverse behaviors whereas our learned policies can be calibrated for up to 1024 distinct style combinations.

References in corpus (22)

• Auto-Encoding Variational Bayes
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• End to End Learning for Self-Driving Cars
• Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations
• Disentangling by Factorising
• A Neural Representation of Sketch Drawings
• DeepMind Control Suite
• Fast and Furious: Real Time End-to-End 3D Detection, Tracking and Motion Forecasting with a Single Convolutional Net
• Character Controllers Using Motion VAEs
• Composing graphical models with neural networks for structured representations and fast inference
• Argoverse: 3D Tracking and Forecasting with Rich Maps
• Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees
• Learning the Structure of Generative Models without Labeled Data
• Representation Learning: A Review and New Perspectives
• Self-Consistent Trajectory Autoencoder: Hierarchical Reinforcement Learning with Trajectory Embeddings
• Generating Long-term Trajectories Using Deep Hierarchical Networks
• Learning Latent Subspaces in Variational Autoencoders
• Coordinated Multi-Agent Imitation Learning
• Generating Multi-Agent Trajectories using Programmatic Weak Supervision
• Learning recurrent representations for hierarchical behavior modeling
• The Atari Grand Challenge Dataset
• Adversarial Information Factorization