Extracting Accurate Materials Data from Research Papers with Conversational Language Models and Prompt Engineering

Abstract

There has been a growing effort to replace manual extraction of data from research papers with automated data extraction based on natural language processing, language models, and recently, large language models (LLMs). Although these methods enable efficient extraction of data from large sets of research papers, they require a significant amount of up-front effort, expertise, and coding. In this work we propose the ChatExtract method that can fully automate very accurate data extraction with minimal initial effort and background, using an advanced conversational LLM. ChatExtract consists of a set of engineered prompts applied to a conversational LLM that both identify sentences with data, extract that data, and assure the data's correctness through a series of follow-up questions. These follow-up questions largely overcome known issues with LLMs providing factually inaccurate responses. ChatExtract can be applied with any conversational LLMs and yields very high quality data extraction. In tests on materials data we find precision and recall both close to 90% from the best conversational LLMs, like ChatGPT-4. We demonstrate that the exceptional performance is enabled by the information retention in a conversational model combined with purposeful redundancy and introducing uncertainty through follow-up prompts. These results suggest that approaches similar to ChatExtract, due to their simplicity, transferability, and accuracy are likely to become powerful tools for data extraction in the near future. Finally, databases for critical cooling rates of metallic glasses and yield strengths of high entropy alloys are developed using ChatExtract.

arXiv comment7 pages, 3 figures, 1 table

References in corpus (12)

• Training language models to follow instructions with human feedback
• LLaMA: Open and Efficient Foundation Language Models
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• Opportunities and Challenges for Machine Learning in Materials Science
• Extracting Accurate Materials Data from Research Papers with Conversational Language Models and Prompt Engineering
• How is ChatGPT's behavior changing over time?
• Machine-learning rationalization and prediction of solid-state synthesis conditions
• Structured information extraction from complex scientific text with fine-tuned large language models
• Database, Features, and Machine Learning Model to Identify Thermally Driven Metal-Insulator Transition Compounds
• A rule-free workflow for the automated generation of databases from scientific literature
• Machine Learning Prediction of Critical Cooling Rate for Metallic Glasses From Expanded Datasets and Elemental Features
• Deliberate then Generate: Enhanced Prompting Framework for Text Generation

Cited by in corpus (13)

• Extracting Accurate Materials Data from Research Papers with Conversational Language Models and Prompt Engineering
• Flexible, Model-Agnostic Method for Materials Data Extraction from Text Using General Purpose Language Models
• Artificial Intelligence in Materials Science and Engineering: Current Landscape, Key Challenges, and Future Trajectorie
• Evaluating Search Engines and Large Language Models for Answering Health Questions
• Exploring the Potential of Large Language Models for Improving Digital Forensic Investigation Efficiency
• The Northeast Materials Database for Magnetic Materials
• DiffractGPT: Atomic Structure Determination from X-ray Diffraction Patterns using Generative Pre-trained Transformer
• Towards an automated workflow in materials science for combining multi-modal simulative and experimental information using data mining and large language models
• Automated Extraction of Material Properties using LLM-based AI Agents
• Machine learning metallic glass critical cooling rates through elemental and molecular simulation based featurization
• Visual Analysis of LLM-based Entity Resolution from Scientific Papers
• Structured Extraction of Process Structure Properties Relationships in Materials Science
• KD-Judge: A Knowledge-Driven Automated Judge Framework for Functional Fitness Movements on Edge Devices