StaQC: A Systematically Mined Question-Code Dataset from Stack Overflow

Abstract

Stack Overflow (SO) has been a great source of natural language questions and their code solutions (i.e., question-code pairs), which are critical for many tasks including code retrieval and annotation. In most existing research, question-code pairs were collected heuristically and tend to have low quality. In this paper, we investigate a new problem of systematically mining question-code pairs from Stack Overflow (in contrast to heuristically collecting them). It is formulated as predicting whether or not a code snippet is a standalone solution to a question. We propose a novel Bi-View Hierarchical Neural Network which can capture both the programming content and the textual context of a code snippet (i.e., two views) to make a prediction. On two manually annotated datasets in Python and SQL domain, our framework substantially outperforms heuristic methods with at least 15% higher F1 and accuracy. Furthermore, we present StaQC (Stack Overflow Question-Code pairs), the largest dataset to date of ~148K Python and ~120K SQL question-code pairs, automatically mined from SO using our framework. Under various case studies, we demonstrate that StaQC can greatly help develop data-hungry models for associating natural language with programming language.

arXiv commentAccepted to the Web Conference 2018 (former WWW 2018), 11 pages, 6 figures

References in corpus (16)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Scikit-learn: Machine Learning in Python
• Distributed Representations of Words and Phrases and their Compositionality
• Neural Machine Translation by Jointly Learning to Align and Translate
• Teaching Machines to Read and Comprehend
• NLTK: The Natural Language Toolkit
• Data Programming: Creating Large Training Sets, Quickly
• Convolutional Neural Networks over Tree Structures for Programming Language Processing
• A Hierarchical Neural Autoencoder for Paragraphs and Documents
• A Convolutional Attention Network for Extreme Summarization of Source Code
• SWIM: Synthesizing What I Mean
• A Syntactic Neural Model for General-Purpose Code Generation
• Latent Predictor Networks for Code Generation
• From Query to Usable Code: An Analysis of Stack Overflow Code Snippets
• A parallel corpus of Python functions and documentation strings for automated code documentation and code generation

Cited by in corpus (18)

• CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation
• CoaCor: Code Annotation for Code Retrieval with Reinforcement Learning
• ProtTrans: Towards Cracking the Language of Life's Code Through Self-Supervised Deep Learning and High Performance Computing
• Leveraging Code Generation to Improve Code Retrieval and Summarization via Dual Learning
• Learning to Mine Aligned Code and Natural Language Pairs from Stack Overflow
• Survey of Code Search Based on Deep Learning
• A Survey on Semantic Parsing
• Neural Code Search Revisited: Enhancing Code Snippet Retrieval through Natural Language Intent
• Optimising the Fit of Stack Overflow Code Snippets into Existing Code
• A Survey on Machine Learning Techniques for Source Code Analysis
• The Good, the Bad, and the Missing: Neural Code Generation for Machine Learning Tasks
• CoNCRA: A Convolutional Neural Network Code Retrieval Approach
• OCoR: An Overlapping-Aware Code Retriever
• Generating Code with the Help of Retrieved Template Functions and Stack Overflow Answers
• COSEA: Convolutional Code Search with Layer-wise Attention
• Rethinking complexity for software code structures: A pioneering study on Linux kernel code repository
• CoSQA: 20,000+ Web Queries for Code Search and Question Answering
• A Comparison of Code Embeddings and Beyond