ThoughtSource: A central hub for large language model reasoning data

Abstract

Large language models (LLMs) such as GPT-4 have recently demonstrated impressive results across a wide range of tasks. LLMs are still limited, however, in that they frequently fail at complex reasoning, their reasoning processes are opaque, they are prone to 'hallucinate' facts, and there are concerns about their underlying biases. Letting models verbalize reasoning steps as natural language, a technique known as chain-of-thought prompting, has recently been proposed as a way to address some of these issues. Here we present ThoughtSource, a meta-dataset and software library for chain-of-thought (CoT) reasoning. The goal of ThoughtSource is to improve future artificial intelligence systems by facilitating qualitative understanding of CoTs, enabling empirical evaluations, and providing training data. This first release of ThoughtSource integrates seven scientific/medical, three general-domain and five math word question answering datasets.

Revision: added datasets, formatting

References in corpus (18)

• Training language models to follow instructions with human feedback
• Evaluating Large Language Models Trained on Code
• Scaling Instruction-Finetuned Language Models
• Large Language Models are Zero-Shot Reasoners
• ReAct: Synergizing Reasoning and Acting in Language Models
• The Pile: An 800GB Dataset of Diverse Text for Language Modeling
• Least-to-Most Prompting Enables Complex Reasoning in Large Language Models
• Can large language models reason about medical questions?
• Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP
• ThoughtSource: A central hub for large language model reasoning data
• Towards Reasoning in Large Language Models: A Survey
• Measuring and Narrowing the Compositionality Gap in Language Models
• Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions
• DERA: Enhancing Large Language Model Completions with Dialog-Enabled Resolving Agents
• BigBIO: A Framework for Data-Centric Biomedical Natural Language Processing
• An automatically discovered chain-of-thought prompt generalizes to novel models and datasets
• Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanations
• Successive Prompting for Decomposing Complex Questions

Cited by in corpus (1)

• ThoughtSource: A central hub for large language model reasoning data