The Differentiable Cross-Entropy Method
arXiv:1909.12830
Abstract
We study the cross-entropy method (CEM) for the non-convex optimization of a continuous and parameterized objective function and introduce a differentiable variant that enables us to differentiate the output of CEM with respect to the objective function's parameters. In the machine learning setting this brings CEM inside of the end-to-end learning pipeline where this has otherwise been impossible. We show applications in a synthetic energy-based structured prediction task and in non-convex continuous control. In the control setting we show how to embed optimal action sequences into a lower-dimensional space. DCEM enables us to fine-tune CEM-based controllers with policy optimization.
arXiv commentICML 2020
References in corpus (45)
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- The NumPy array: a structure for efficient numerical computation
- Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
- Stochastic Backpropagation and Approximate Inference in Deep Generative Models
- Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
- DeepMind Control Suite
- Neural Architecture Optimization
- Unrolled Generative Adversarial Networks
- Continuous-Time Gaussian Process Motion Planning via Probabilistic Inference
- Composing graphical models with neural networks for structured representations and fast inference
- Meta-Learning with Latent Embedding Optimization
- Differentiable MPC for End-to-end Planning and Control
- Input Convex Neural Networks
- Fast Context Adaptation via Meta-Learning
- Learning to Learn without Gradient Descent by Gradient Descent
- Learning to Optimize
- Temporal Difference Models: Model-Free Deep RL for Model-Based Control
- Differentiable Convex Optimization Layers
- Hyperparameter optimization with approximate gradient
- SOLAR: Deep Structured Representations for Model-Based Reinforcement Learning
- Task-based End-to-end Model Learning in Stochastic Optimization
- On Differentiating Parameterized Argmin and Argmax Problems with Application to Bi-level Optimization
- Learning Plannable Representations with Causal InfoGAN
- Universal Planning Networks
- Monte Carlo Gradient Estimation in Machine Learning
- Self-Consistent Trajectory Autoencoder: Hierarchical Reinforcement Learning with Trajectory Embeddings
- Exploring Model-based Planning with Policy Networks
- BOCK : Bayesian Optimization with Cylindrical Kernels
- DeepMDP: Learning Continuous Latent Space Models for Representation Learning
- Deep Set Prediction Networks
- Adaptive and Safe Bayesian Optimization in High Dimensions via One-Dimensional Subspaces
- Path Integral Networks: End-to-End Differentiable Optimal Control
- Variational Inference MPC for Bayesian Model-based Reinforcement Learning
- Design by adaptive sampling
- Goal-Driven Dynamics Learning via Bayesian Optimization
- Active Learning of Linear Embeddings for Gaussian Processes
- Robust Locally-Linear Controllable Embedding
- Learning Latent Plans from Play
- Objective Mismatch in Model-based Reinforcement Learning
- Disentangled State Space Representations
- The Limited Multi-Label Projection Layer
- Bayesian Optimization in Variational Latent Spaces with Dynamic Compression
- Meta-Learning Acquisition Functions for Transfer Learning in Bayesian Optimization
- Learning Convex Optimization Control Policies
- Prediction, Consistency, Curvature: Representation Learning for Locally-Linear Control