27 citations · 81 across the 33 of their papers we have counts for
35 papers
The Canonical Order Problem: When Large Language Models Are Unreliable Knowledge Bases for Multi-Valued Relations
Timo Pierre Schrader, Annemarie Friedrich, Simon Razniewski +1
Large language models (LLMs) are increasingly used as knowledge bases (KBs) due to the vast amount of knowledge they acquire during pre-training. While many works focus on extracti…
GPTKB 2.0: Browsing, Querying, and Auditing a Disambiguated LLM-Derived Knowledge Base
Yujia Hu, Tuan-Phong Nguyen, Simon Razniewski
We present a web demo for exploring a large-scale disambiguated knowledge base (KB) materialized from a large language model (LLM). GPTKB 2.0 contains 38.4M triples over 1.6M canon…
Constructing Disambiguated Knowledge Bases from Large Language Models at Scale
Yujia Hu, Tuan-Phong Nguyen, Simon Razniewski
Automated Knowledge Base Construction (AKBC) is a core NLP task, and recent work proposes generating knowledge bases directly from large language models (LLMs), treating the model…
LLMPEDIA: Browsing, Verifying, and Comparing the Parametric Encyclopedic Knowledge of LLMs
Muhammed Saeed, Simon Razniewski
Flagship language models appear saturated on benchmarks like MMLU (Hendrycks et al., 2021), scoring above 90% - yet benchmarks test only what the experimenter thought to ask, the a…
Is It Novel and Why? Fine-Grained Patent Novelty Prediction Based on Passage Retrieval
Valentin Knappich, Anna Hätty, Simon Razniewski +1
Novelty assessment is a critical yet complex task in the examination process for patent acceptance, requiring examiners to determine whether an invention is disclosed in a prior ar…
Beyond Questions: Evaluating the Stated Knowledge of LLMs
Luca Giordano, Simon Razniewski
Parametric knowledge in large language models (LLMs) is a cornerstone of their success, yet remains poorly understood. Existing knowledge benchmarks typically rely on predefined qu…
LLMpedia: A Transparent Framework to Materialize an LLM's Encyclopedic Knowledge at Scale
Muhammed Saeed, Simon Razniewski
Benchmarks like MMLU suggest flagship language models approach factuality saturation above 90\%. \emph{LLMpedia} shows this picture is incomplete. We materialize 1.3M encyc…
A Solver-in-the-Loop Framework for Improving LLMs on Answer Set Programming for Logic Puzzle Solving
Timo Pierre Schrader, Lukas Lange, Tobias Kaminski +2
The rise of large language models (LLMs) has sparked interest in coding assistants. While general-purpose programming languages are well supported, generating code for domain-speci…
Auditing a KB Elicitation of Frontier LLM Knowledge: A Multi-dimensional Analysis of GPTKB v1.5
Shrestha Ghosh, Luca Giordano, Yujia Hu +2
LLMs are remarkable artifacts that have revolutionized a range of knowledge-intensive tasks. A significant contributor is their factual knowledge, which, to date, remains poorly un…
Foundations of LLM Knowledge Materialization: Termination, Reproducibility, Robustness
Luca Giordano, Simon Razniewski
Large Language Models (LLMs) encode substantial factual knowledge, yet measuring and systematizing this knowledge remains challenging. Converting it into structured format, for exa…
AGENTS-LLM: Augmentative GENeration of Challenging Traffic Scenarios with an Agentic LLM Framework
Yu Yao, Salil Bhatnagar, Markus Mazzola +5
Rare, yet critical, scenarios pose a significant challenge in testing and evaluating autonomous driving planners. Relying solely on real-world driving scenes requires collecting ma…
GPTKB v1.5: A Massive Knowledge Base for Exploring Factual LLM Knowledge
Yujia Hu, Tuan-Phong Nguyen, Shrestha Ghosh +2
Language models are powerful artifacts, yet their factual knowledge is still poorly understood, and inaccessible to ad-hoc browsing and scalable statistical analysis. This demonstr…
PEDANTIC: A Dataset for the Automatic Examination of Definiteness in Patent Claims
Valentin Knappich, Annemarie Friedrich, Anna Hätty +1
Patent claims define the scope of protection for an invention. If there are ambiguities in a claim, it is rejected by the patent office. In the US, this is referred to as indefinit…
Enabling LLM Knowledge Analysis via Extensive Materialization
Yujia Hu, Tuan-Phong Nguyen, Shrestha Ghosh +1
Large language models (LLMs) have majorly advanced NLP and AI, and next to their ability to perform a wide range of procedural tasks, a major success factor is their internalized f…
QUITE: Quantifying Uncertainty in Natural Language Text in Bayesian Reasoning Scenarios
Timo Pierre Schrader, Lukas Lange, Simon Razniewski +1
Reasoning is key to many decision making processes. It requires consolidating a set of rule-like premises that are often associated with degrees of uncertainty and observations to…
Pap2Pat: Benchmarking Outline-Guided Long-Text Patent Generation with Patent-Paper Pairs
Valentin Knappich, Simon Razniewski, Anna Hätty +1
Dealing with long and highly complex technical text is a challenge for Large Language Models (LLMs), which still have to unfold their potential in supporting expensive and timeinte…
Recall Them All: Retrieval-Augmented Language Models for Long Object List Extraction from Long Documents
Sneha Singhania, Simon Razniewski, Gerhard Weikum
Methods for relation extraction from text mostly focus on high precision, at the cost of limited recall. High recall is crucial, though, to populate long lists of object entities t…
Cultural Commonsense Knowledge for Intercultural Dialogues
Tuan-Phong Nguyen, Simon Razniewski, Gerhard Weikum
Despite recent progress, large language models (LLMs) still face the challenge of appropriately reacting to the intricacies of social and cultural conventions. This paper presents…
BoschAI @ Causal News Corpus 2023: Robust Cause-Effect Span Extraction using Multi-Layer Sequence Tagging and Data Augmentation
Timo Pierre Schrader, Simon Razniewski, Lukas Lange +1
Understanding causality is a core aspect of intelligence. The Event Causality Identification with Causal News Corpus Shared Task addresses two aspects of this challenge: Subtask 1…
BoschAI @ PLABA 2023: Leveraging Edit Operations in End-to-End Neural Sentence Simplification
Valentin Knappich, Simon Razniewski, Annemarie Friedrich
Automatic simplification can help laypeople to comprehend complex scientific text. Language models are frequently applied to this task by translating from complex to simple languag…
Evaluating the Knowledge Base Completion Potential of GPT
Blerta Veseli, Simon Razniewski, Jan-Christoph Kalo +1
Structured knowledge bases (KBs) are an asset for search engines and other applications, but are inevitably incomplete. Language models (LMs) have been proposed for unsupervised kn…
Large Language Models and Knowledge Graphs: Opportunities and Challenges
Jeff Z. Pan, Simon Razniewski, Jan-Christoph Kalo +13
Large Language Models (LLMs) have taken Knowledge Representation -- and the world -- by storm. This inflection point marks a shift from explicit knowledge representation to a renew…
Extracting Multi-valued Relations from Language Models
Sneha Singhania, Simon Razniewski, Gerhard Weikum
The widespread usage of latent language representations via pre-trained language models (LMs) suggests that they are a promising source of structured knowledge. However, existing m…
Can large language models generate salient negative statements?
Hiba Arnaout, Simon Razniewski
We examine the ability of large language models (LLMs) to generate salient (interesting) negative statements about real-world entities; an emerging research topic of the last few y…