Textbooks Are All You Need

Abstract

We introduce phi-1, a new large language model for code, with significantly smaller size than competing models: phi-1 is a Transformer-based model with 1.3B parameters, trained for 4 days on 8 A100s, using a selection of ``textbook quality" data from the web (6B tokens) and synthetically generated textbooks and exercises with GPT-3.5 (1B tokens). Despite this small scale, phi-1 attains pass@1 accuracy 50.6% on HumanEval and 55.5% on MBPP. It also displays surprising emergent properties compared to phi-1-base, our model before our finetuning stage on a dataset of coding exercises, and phi-1-small, a smaller model with 350M parameters trained with the same pipeline as phi-1 that still achieves 45% on HumanEval.

arXiv comment26 pages; changed color scheme of plot. fixed minor typos and added couple clarifications

Cited by in corpus (9)

• Quo Vadis ChatGPT? From Large Language Models to Large Knowledge Models
• Metamorphic Malware Evolution: The Potential and Peril of Large Language Models
• Tucano: Advancing Neural Text Generation for Portuguese
• Insights into resource utilization of code small language models serving with runtime engines and execution providers
• Cracks in The Stack: Hidden Vulnerabilities and Licensing Risks in LLM Pre-Training Datasets
• Fabricator: An Open Source Toolkit for Generating Labeled Training Data with Teacher LLMs
• Hallucinations in LLMs: A Lifecycle-Based Survey of Causes, Detection, Mitigation, and Prevention
• A Glitch in the Matrix? Locating and Detecting Language Model Grounding with Fakepedia
• AI-generated text boundary detection with RoFT