Visual Analysis of LLM-based Entity Resolution from Scientific Papers
arXiv:2609.06037 · doi:10.1016/j.visinf.2025.100236
Abstract
This paper focuses on the visual analytics support for extracting domain-specific entity from extensive scientific literature, a task with inherent limitations using traditional named entity resolution methods. With the advent of large language models (LLMs) such as GPT-4, significant improvements over conventional machine learning approaches have been achieved due to LLM's capability on entity resolution integrate abilities such as understanding multiple types of text. This research introduces a new visual analysis pipeline that integrates these advanced LLMs with versatile visualization and interaction designs to support batch entity resolution. Specifically, we focus on a specific material science field of Metal-Organic Frameworks (MOFs) and a large data collection namely CSD-MOFs. Through collaboration with domain experts in material science, we obtain well-labeled synthesis paragraphs. We propose human-in-the-loop refinement over the entity resolution process using visual analytics techniques, which allows domain experts to interactively integrate insights into LLM intelligence, including error analysis and interpretation of the retrieval-augmented generation (RAG) algorithm. Our evaluation through the case study of example selection for RAG demonstrates that this human-machine collaborative approach improved single-document entity resolution accuracy by approximately 30%.
arXiv comment10 pages, 8 figures
References in corpus (18)
- Llama 2: Open Foundation and Fine-Tuned Chat Models
- A Survey on Large Language Model based Autonomous Agents
- Sparks of Artificial General Intelligence: Early experiments with GPT-4
- A Survey on Deep Learning for Named Entity Recognition
- Gemini: A Family of Highly Capable Multimodal Models
- A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT
- ChatGPT Chemistry Assistant for Text Mining and Prediction of MOF Synthesis
- Extracting Accurate Materials Data from Research Papers with Conversational Language Models and Prompt Engineering
- GPT-NER: Named Entity Recognition via Large Language Models
- Structured information extraction from complex scientific text with fine-tuned large language models
- The Impact of Large Language Models on Scientific Discovery: a Preliminary Study using GPT-4
- Principled Instructions Are All You Need for Questioning LLaMA-1/2, GPT-3.5/4
- Flexible, Model-Agnostic Method for Materials Data Extraction from Text Using General Purpose Language Models
- LightVA: Lightweight Visual Analytics with LLM Agent-Based Task Planning and Execution
- embComp: Visual Interactive Comparison of Vector Embeddings
- Code Generation with AlphaCodium: From Prompt Engineering to Flow Engineering
- ChatMOF: An Autonomous AI System for Predicting and Generating Metal-Organic Frameworks
- LLM-based MOFs Synthesis Condition Extraction using Few-Shot Demonstrations