On Adversarial Robustness of Synthetic Code Generation
arXiv:2106.11629
Abstract
Automatic code synthesis from natural language descriptions is a challenging task. We witness massive progress in developing code generation systems for domain-specific languages (DSLs) employing sequence-to-sequence deep learning techniques in the recent past. In this paper, we specifically experiment with \textsc{AlgoLisp} DSL-based generative models and showcase the existence of significant dataset bias through different classes of adversarial examples. We also experiment with two variants of Transformer-based models that outperform all existing \textsc{AlgoLisp} DSL-based code generation baselines. Consistent with the current state-of-the-art systems, our proposed models, too, achieve poor performance under adversarial settings. Therefore, we propose several dataset augmentation techniques to reduce bias and showcase their efficacy using robust experimentation.
References in corpus (8)
- Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning
- DeepCoder: Learning to Write Programs
- RobustFill: Neural Program Learning under Noisy I/O
- Leveraging Grammar and Reinforcement Learning for Neural Program Synthesis
- Learning to Infer Program Sketches
- Neural Program Search: Solving Programming Tasks from Description and Examples
- NAPS: Natural Program Synthesis Dataset
- Ain't Nobody Got Time For Coding: Structure-Aware Program Synthesis From Natural Language