Galactica: A Large Language Model for Science

Abstract

Information overload is a major obstacle to scientific progress. The explosive growth in scientific literature and data has made it ever harder to discover useful insights in a large mass of information. Today scientific knowledge is accessed through search engines, but they are unable to organize scientific knowledge alone. In this paper we introduce Galactica: a large language model that can store, combine and reason about scientific knowledge. We train on a large scientific corpus of papers, reference material, knowledge bases and many other sources. We outperform existing models on a range of scientific tasks. On technical knowledge probes such as LaTeX equations, Galactica outperforms the latest GPT-3 by 68.2% versus 49.0%. Galactica also performs well on reasoning, outperforming Chinchilla on mathematical MMLU by 41.3% to 35.7%, and PaLM 540B on MATH with a score of 20.4% versus 8.8%. It also sets a new state-of-the-art on downstream tasks such as PubMedQA and MedMCQA dev of 77.6% and 52.9%. And despite not being trained on a general corpus, Galactica outperforms BLOOM and OPT-175B on BIG-bench. We believe these results demonstrate the potential for language models as a new interface for science. We open source the model for the benefit of the scientific community.

Cited by in corpus (21)

• 14 Examples of How LLMs Can Transform Materials Science and Chemistry: A Reflection on a Large Language Model Hackathon
• Cyberbullying Detection for Low-resource Languages and Dialects: Review of the State of the Art
• When Geoscience Meets Generative AI and Large Language Models: Foundations, Trends, and Future Challenges
• Synergi: A Mixed-Initiative System for Scholarly Synthesis and Sensemaking
• A Survey on Symbolic Knowledge Distillation of Large Language Models
• Achieving Peak Performance for Large Language Models: A Systematic Review
• nach0: Multimodal Natural and Chemical Languages Foundation Model
• Exploring the Frontiers of LLMs in Psychological Applications: A Comprehensive Review
• Developing ChemDFM as a large language foundation model for chemistry
• Inferring pointwise diffusion properties of single trajectories with deep learning
• An Evaluation on Large Language Model Outputs: Discourse and Memorization
• Domain-specific ChatBots for Science using Embeddings
• Spiers Memorial Lecture: How to do impactful research in artificial intelligence for chemistry and materials science
• GPT Struct Me: Probing GPT Models on Narrative Entity Extraction
• Enhancing chest X-ray datasets with privacy-preserving large language models and multi-type annotations: a data-driven approach for improved classification
• Complex QA and language models hybrid architectures, Survey
• Large Language Models and Multimodal Retrieval for Visual Word Sense Disambiguation
• AutoMathKG: The automated mathematical knowledge graph based on LLM and vector database
• Variational Open-Domain Question Answering
• Leveraging Knowledge Graph Embeddings to Enhance Contextual Representations for Relation Extraction
• Emergent inabilities? Inverse scaling over the course of pretraining