Energy-Based Models for Code Generation under Compilability Constraints
arXiv:2106.04985
Abstract
Neural language models can be successfully trained on source code, leading to applications such as code completion. However, their versatile autoregressive self-supervision objective overlooks important global sequence-level features that are present in the data such as syntactic correctness or compilability. In this work, we pose the problem of learning to generate compilable code as constraint satisfaction. We define an Energy-Based Model (EBM) representing a pre-trained generative model with an imposed constraint of generating only compilable sequences. We then use the KL-Adaptive Distributional Policy Gradient algorithm (Khalifa et al., 2021) to train a generative model approximating the EBM. We conduct experiments showing that our proposed approach is able to improve compilability rates without sacrificing diversity and complexity of the generated samples.
Accepted for the First Workshop on Natural Language Processing for Programming, ACL 2021
References in corpus (7)
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning
- CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation
- Unsupervised Translation of Programming Languages
- ENGINE: Energy-Based Inference Networks for Non-Autoregressive Machine Translation
- Distributional Reinforcement Learning for Energy-Based Sequential Models
- Code Completion using Neural Attention and Byte Pair Encoding