CodeSearchNet Challenge: Evaluating the State of Semantic Code Search

Abstract

Semantic code search is the task of retrieving relevant code given a natural language query. While related to other information retrieval tasks, it requires bridging the gap between the language used in code (often abbreviated and highly technical) and natural language more suitable to describe vague concepts and ideas. To enable evaluation of progress on code search, we are releasing the CodeSearchNet Corpus and are presenting the CodeSearchNet Challenge, which consists of 99 natural language queries with about 4k expert relevance annotations of likely results from CodeSearchNet Corpus. The corpus contains about 6 million functions from open-source code spanning six programming languages (Go, Java, JavaScript, PHP, Python, and Ruby). The CodeSearchNet Corpus also contains automatically generated query-like natural language for 2 million functions, obtained from mechanically scraping and preprocessing associated function documentation. In this article, we describe the methodology used to obtain the corpus and expert labels, as well as a number of simple baseline solutions for the task. We hope that CodeSearchNet Challenge encourages researchers and practitioners to study this interesting task further and will host a competition and leaderboard to track the progress on the challenge. We are also keen on extending CodeSearchNet Challenge to more queries and programming languages in the future.

arXiv commentUpdated evaluation numbers after fixing indexing bug

Cited by in corpus (36)

• Practical Program Repair in the Era of Large Pre-trained Language Models
• Natural Attack for Pre-trained Models of Code
• No More Fine-Tuning? An Experimental Evaluation of Prompt Tuning in Code Intelligence
• Natural Language Generation and Understanding of Big Code for AI-Assisted Programming: A Review
• On the Feasibility of Transfer-learning Code Smells using Deep Learning
• Generating Accurate Assert Statements for Unit Test Cases using Pretrained Transformers
• What Makes Good In-context Demonstrations for Code Intelligence Tasks with LLMs?
• Semantic Robustness of Models of Source Code
• Code Search based on Context-aware Code Translation
• HPC-GPT: Integrating Large Language Model for High-Performance Computing
• Fully Autonomous Programming with Large Language Models
• Diet Code Is Healthy: Simplifying Programs for Pre-trained Models of Code
• On the Effectiveness of Transfer Learning for Code Search
• Several categories of Large Language Models (LLMs): A Short Survey
• CodeMatcher: Searching Code Based on Sequential Semantics of Important Query Words
• Revisiting the Plastic Surgery Hypothesis via Large Language Models
• Explaining Software Bugs Leveraging Code Structures in Neural Machine Translation
• Can We Generate Shellcodes via Natural Language? An Empirical Study
• Survey of Code Search Based on Deep Learning
• A Comprehensive Review of State-of-The-Art Methods for Java Code Generation from Natural Language Text
• Toward a Theory of Causation for Interpreting Neural Code Models
• T5APR: Empowering Automated Program Repair across Languages through Checkpoint Ensemble
• Exploring and Evaluating Personalized Models for Code Generation
• CS1QA: A Dataset for Assisting Code-based Question Answering in an Introductory Programming Course
• PyTester: Deep Reinforcement Learning for Text-to-Testcase Generation
• Senatus -- A Fast and Accurate Code-to-Code Recommendation Engine
• Lyra: A Benchmark for Turducken-Style Code Generation
• Enhancing Function Name Prediction using Votes-Based Name Tokenization and Multi-Task Learning
• Semantic Code Graph -- an information model to facilitate software comprehension
• Study of Distractors in Neural Models of Code
• Towards Using Data-Influence Methods to Detect Noisy Samples in Source Code Corpora
• Evaluating and Optimizing the Effectiveness of Neural Machine Translation in Supporting Code Retrieval Models: A Study on the CAT Benchmark
• INSPECT: Intrinsic and Systematic Probing Evaluation for Code Transformers
• CoRRPUS: Code-based Structured Prompting for Neurosymbolic Story Understanding
• UNICS: Multilingual Code Search via Unified Pseudocode and Contrastive Transfer Learning
• MultiMend: Multilingual Program Repair with Context Augmentation and Multi-Hunk Patch Generation