Human Evaluation of Procedural Knowledge Graph Extraction from Text with Large Language Models
arXiv:2412.03589 · doi:10.1007/978-3-031-77792-9_26
Abstract
Procedural Knowledge is the know-how expressed in the form of sequences of steps needed to perform some tasks. Procedures are usually described by means of natural language texts, such as recipes or maintenance manuals, possibly spread across different documents and systems, and their interpretation and subsequent execution is often left to the reader. Representing such procedures in a Knowledge Graph (KG) can be the basis to build digital tools to support those users who need to apply or execute them. In this paper, we leverage Large Language Model (LLM) capabilities and propose a prompt engineering approach to extract steps, actions, objects, equipment and temporal information from a textual procedure, in order to populate a Procedural KG according to a pre-defined ontology. We evaluate the KG extraction results by means of a user study, in order to qualitatively and quantitatively assess the perceived quality and usefulness of the LLM-extracted procedural knowledge. We show that LLMs can produce outputs of acceptable quality and we assess the subjective perception of AI by human evaluators.
References in corpus (9)
- Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Unifying Large Language Models and Knowledge Graphs: A Roadmap
- AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback
- CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotation
- Reasoning about Goals, Steps, and Temporal Ordering with WikiHow
- OntoChat: a Framework for Conversational Ontology Engineering using Language Models
- Reasoning about Procedures with Natural Language Processing: A Tutorial
- Using Large Language Models for Knowledge Engineering (LLMKE): A Case Study on Wikidata
- Expanding the Vocabulary of BERT for Knowledge Base Construction