StarCoder: may the source be with you!

Abstract

The BigCode community, an open-scientific collaboration working on the responsible development of Large Language Models for Code (Code LLMs), introduces StarCoder and StarCoderBase: 15.5B parameter models with 8K context length, infilling capabilities and fast large-batch inference enabled by multi-query attention. StarCoderBase is trained on 1 trillion tokens sourced from The Stack, a large collection of permissively licensed GitHub repositories with inspection tools and an opt-out process. We fine-tuned StarCoderBase on 35B Python tokens, resulting in the creation of StarCoder. We perform the most comprehensive evaluation of Code LLMs to date and show that StarCoderBase outperforms every open Code LLM that supports multiple programming languages and matches or outperforms the OpenAI code-cushman-001 model. Furthermore, StarCoder outperforms every model that is fine-tuned on Python, can be prompted to achieve 40\% pass@1 on HumanEval, and still retains its performance on other programming languages. We take several important steps towards a safe open-access model release, including an improved PII redaction pipeline and a novel attribution tracing tool, and make the StarCoder models publicly available under a more commercially viable version of the Open Responsible AI Model license.

Cited by in corpus (26)

• Large Language Models for Code: Security Hardening and Adversarial Testing
• Natural Language Generation and Understanding of Big Code for AI-Assisted Programming: A Review
• LLM-Based Test-Driven Interactive Code Generation: User Study and Empirical Evaluation
• RTLCoder: Outperforming GPT-3.5 in Design RTL Generation with Our Open-Source Dataset and Lightweight Solution
• LLM for SoC Security: A Paradigm Shift
• A Performance Study of LLM-Generated Code on Leetcode
• The Dawn of Natural Language to SQL: Are We Fully Ready?
• Refactoring Programs Using Large Language Models with Few-Shot Examples
• LINC: A Neurosymbolic Approach for Logical Reasoning by Combining Language Models with First-Order Logic Provers
• Deploying and Evaluating LLMs to Program Service Mobile Robots
• CoderEval: A Benchmark of Pragmatic Code Generation with Generative Pre-trained Models
• TR2MTL: LLM based framework for Metric Temporal Logic Formalization of Traffic Rules
• Exploring the Potential of Large Language Models for Improving Digital Forensic Investigation Efficiency
• Output Format Biases in the Evaluation of Large Language Models for Code Translation
• Out of the Cage: How Stochastic Parrots Win in Cyber Security Environments
• The EarlyBIRD Catches the Bug: On Exploiting Early Layers of Encoder Models for More Efficient Code Classification
• Metamorphic Malware Evolution: The Potential and Peril of Large Language Models
• PyTester: Deep Reinforcement Learning for Text-to-Testcase Generation
• Detect Llama -- Finding Vulnerabilities in Smart Contracts using Large Language Models
• LLMs and Stack Overflow Discussions: Reliability, Impact, and Challenges
• Program Repair with Minimal Edits Using CodeT5
• Large Language Models for Combinatorial Optimization: A Systematic Review
• Retrieval-augmented code completion for local projects using large language models
• CoEdPilot: Recommending Code Edits with Learned Prior Edit Relevance, Project-wise Awareness, and Interactive Nature
• API-Assisted Code Generation for Question Answering on Varied Table Structures
• SEED: Enhancing Text-to-SQL Performance and Practical Usability Through Automatic Evidence Generation