4 papers
UsefulBench: Towards Decision-Useful Information as a Target for Information Retrieval
Tobias Schimanski, Stefanie Lewandowski, Christian Woerle +3
Conventional information retrieval is concerned with identifying the relevance of texts for a given query. Yet, the conventional definition of relevance is dominated by aspects of…
pdfQA: Diverse, Challenging, and Realistic Question Answering over PDFs
Tobias Schimanski, Imene Kolli, Yu Fan +4
PDFs are the second-most used document type on the internet (after HTML). Yet, existing QA datasets commonly start from text sources or only address specific domains. In this paper…
DIRAS: Efficient LLM Annotation of Document Relevance in Retrieval Augmented Generation
Jingwei Ni, Tobias Schimanski, Meihong Lin +3
Retrieval Augmented Generation (RAG) is widely employed to ground responses to queries on domain-specific documents. But do RAG implementations leave out important information when…
ClimRetrieve: A Benchmarking Dataset for Information Retrieval from Corporate Climate Disclosures
Tobias Schimanski, Jingwei Ni, Roberto Spacey +2
To handle the vast amounts of qualitative data produced in corporate climate communication, stakeholders increasingly rely on Retrieval Augmented Generation (RAG) systems. However,…