Annotative Indexing

Abstract

This paper introduces annotative indexing, a novel framework that unifies and generalizes traditional inverted indexes, column stores, object stores, and graph databases. As a result, annotative indexing can provide the underlying indexing framework for databases that support retrieval augmented generation, knowledge graphs, entity retrieval, semi-structured data, and ranked retrieval. While we primarily focus on human language data in the form of text, annotative indexing is sufficiently general to support a range of other datatypes, and we provide examples of SQL-like queries over a JSON store that includes numbers and dates. Taking advantage of the flexibility of annotative indexing, we also demonstrate a fully dynamic annotative index incorporating support for ACID properties of transactions with hundreds of multiple concurrent readers and writers.

Code at https://github.com/claclark/Cottontail

References in corpus (7)

• Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs
• Distilling Dense Representations for Ranking using Tightly-Coupled Teachers
• Efficient Inverted Indexes for Approximate Retrieval over Learned Sparse Representations
• Dense Text Retrieval based on Pretrained Language Models: A Survey
• Efficient Neural Ranking using Forward Indexes
• Survey of Vector Database Management Systems
• Faster Learned Sparse Retrieval with Block-Max Pruning