RobustFill: Neural Program Learning under Noisy I/O

Abstract

The problem of automatically generating a computer program from some specification has been studied since the early days of AI. Recently, two competing approaches for automatic program learning have received significant attention: (1) neural program synthesis, where a neural network is conditioned on input/output (I/O) examples and learns to generate a program, and (2) neural program induction, where a neural network generates new outputs directly using a latent program representation. Here, for the first time, we directly compare both approaches on a large-scale, real-world learning task. We additionally contrast to rule-based program synthesis, which uses hand-crafted semantics to guide the program generation. Our neural models use a modified attention RNN to allow encoding of variable-sized sets of I/O pairs. Our best synthesis model achieves 92% accuracy on a real-world test set, compared to the 34% accuracy of the previous best neural synthesis approach. The synthesis model also outperforms a comparable induction model on this task, but we more importantly demonstrate that the strength of each approach is highly dependent on the evaluation metric and end-user application. Finally, we show that we can train our neural models to remain very robust to the type of noise expected in real-world data (e.g., typos), while a highly-engineered rule-based system fails entirely.

8 pages + 9 pages of supplementary material

References in corpus (5)

• Grammar as a Foreign Language
• Inferring Algorithmic Patterns with Stack-Augmented Recurrent Nets
• Neural Turing Machines
• Neuro-Symbolic Program Synthesis
• TerpreT: A Probabilistic Programming Language for Program Induction

Cited by in corpus (17)

• Write, Execute, Assess: Program Synthesis with a REPL
• Learning to Represent Programs with Property Signatures
• SPoC: Search-based Pseudocode to Code
• Neural Inverse Knitting: From Images to Manufacturing Instructions
• Guiding High-Performance SAT Solvers with Unsat-Core Predictions
• Learning to learn generative programs with Memoised Wake-Sleep
• Programmable Agents
• Intelligence, physics and information -- the tradeoff between accuracy and simplicity in machine learning
• Data Generation for Neural Programming by Example
• Benchmarking Multimodal Regex Synthesis with Complex Structures
• Hierarchical Variational Imitation Learning of Control Programs
• Gradient Descent over Metagrammars for Syntax-Guided Synthesis
• PLANS: Robust Program Learning from Neurally Inferred Specifications
• Biological Blueprints for Next Generation AI Systems
• Grammar Filtering For Syntax-Guided Synthesis
• Type-driven Neural Programming by Example
• Formal Fields: A Framework to Automate Code Generation Across Domains