Large Language Models, scientific knowledge and factuality: A framework to streamline human expert evaluation

Abstract

The paper introduces a framework for the evaluation of the encoding of factual scientific knowledge, designed to streamline the manual evaluation process typically conducted by domain experts. Inferring over and extracting information from Large Language Models (LLMs) trained on a large corpus of scientific literature can potentially define a step change in biomedical discovery, reducing the barriers for accessing and integrating existing medical evidence. This work explores the potential of LLMs for dialoguing with biomedical background knowledge, using the context of antibiotic discovery. The framework involves of three evaluation steps, each assessing different aspects sequentially: fluency, prompt alignment, semantic coherence, factual knowledge, and specificity of the generated responses. By splitting these tasks between non-experts and experts, the framework reduces the effort required from the latter. The work provides a systematic assessment on the ability of eleven state-of-the-art models LLMs, including ChatGPT, GPT-4 and Llama 2, in two prompting-based tasks: chemical compound definition generation and chemical compound-fungus relation determination. Although recent models have improved in fluency, factual accuracy is still low and models are biased towards over-represented entities. The ability of LLMs to serve as biomedical knowledge bases is questioned, and the need for additional systematic evaluation frameworks is highlighted. While LLMs are currently not fit for purpose to be used as biomedical factual knowledge bases in a zero-shot setting, there is a promising emerging property in the direction of factuality as the models become domain specialised, scale-up in size and level of human feedback.

Accepted at the Journal of Biomedical Informatics, Volume 158, October 2024, 104724

References in corpus (33)

• Survey of Hallucination in Natural Language Generation
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining
• The Curious Case of Neural Text Degeneration
• ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks
• Capabilities of GPT-4 on Medical Challenge Problems
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• The Pile: An 800GB Dataset of Diverse Text for Language Modeling
• Large Language Models Are Human-Level Prompt Engineers
• ChatGPT-4 Outperforms Experts and Crowd Workers in Annotating Political Twitter Messages with Zero-Shot Learning
• Large Language Models Struggle to Learn Long-Tail Knowledge
• What can Large Language Models do in chemistry? A comprehensive benchmark on eight tasks
• Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets
• How Context Affects Language Models' Factual Predictions
• TrustLLM: Trustworthiness in Large Language Models
• Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models
• Measuring Attribution in Natural Language Generation Models
• A Stitch in Time Saves Nine: Detecting and Mitigating Hallucinations of LLMs by Validating Low-Confidence Generation
• Emergent and Predictable Memorization in Large Language Models
• Benchmarking Cognitive Biases in Large Language Models as Evaluators
• Transformers and the representation of biomedical background knowledge
• Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models
• Style Over Substance: Evaluation Biases for Large Language Models
• Do Transformers Encode a Foundational Ontology? Probing Abstract Classes in Natural Language
• Measuring Reliability of Large Language Models through Semantic Consistency
• Comparative Performance Evaluation of Large Language Models for Extracting Molecular Interactions and Pathway Knowledge
• Hallucination is the last thing you need
• Leveraging Large Language Models for NLG Evaluation: Advances and Challenges
• An Evaluation of Large Language Models in Bioinformatics Research
• An LLM-based Knowledge Synthesis and Scientific Reasoning Framework for Biomedical Discovery
• Pitfalls of Conversational LLMs on News Debiasing
• Understanding the effects of language-specific class imbalance in multilingual fine-tuning
• Relation Extraction in underexplored biomedical domains: A diversity-optimised sampling and synthetic data generation approach