A parallel corpus of Python functions and documentation strings for automated code documentation and code generation
arXiv:1707.02275
Abstract
Automated documentation of programming source code and automated code generation from natural language are challenging tasks of both practical and scientific interest. Progress in these areas has been limited by the low availability of parallel corpora of code and natural language descriptions, which tend to be small and constrained to specific domains. In this work we introduce a large and diverse parallel corpus of a hundred thousands Python functions with their documentation strings ("docstrings") generated by scraping open source repositories on GitHub. We describe baseline results for the code documentation and code generation tasks obtained by neural machine translation. We also experiment with data augmentation techniques to further increase the amount of training data. We release our datasets and processing scripts in order to stimulate research in these areas.
5 pages, 1 figure, 3 tables
References in corpus (3)
Cited by in corpus (25)
- Evaluating Large Language Models Trained on Code
- CodeBLEU: a Method for Automatic Evaluation of Code Synthesis
- Structured Neural Summarization
- Code Structure Guided Transformer for Source Code Summarization
- StaQC: A Systematically Mined Question-Code Dataset from Stack Overflow
- Graph Neural Networks for Natural Language Processing: A Survey
- Retrieval-Augmented Generation for Code Summarization via Hybrid GNN
- Unsupervised Translation of Programming Languages
- Deep Learning Based Chatbot Models
- Neural Program Synthesis with Priority Queue Training
- EVIL: Exploiting Software via Natural Language
- Learning to Mine Aligned Code and Natural Language Pairs from Stack Overflow
- A Survey on Semantic Parsing
- TranS^3: A Transformer-based Framework for Unifying Code Summarization and Code Search
- Polyglot Semantic Parsing in APIs
- A Survey on Machine Learning Techniques for Source Code Analysis
- Leveraging Automated Unit Tests for Unsupervised Code Translation
- GN-Transformer: Fusing Sequence and Graph Representation for Improved Code Summarization
- Code Summarization with Structure-induced Transformer
- Variational Prefix Tuning for Diverse and Accurate Code Summarization Using Pre-trained Language Models
- IOT: Instance-wise Layer Reordering for Transformer Structures
- Exploiting Method Names to Improve Code Summarization: A Deliberation Multi-Task Learning Approach
- Self Learning from Large Scale Code Corpus to Infer Structure of Method Invocations
- Improving Automatic Source Code Summarization via Deep Reinforcement Learning
- CodeQA: A Question Answering Dataset for Source Code Comprehension