5 papers
Closing the Calibration Gap in Semantic Caching
Aditeya Baral, Radoslav Ralev, Iliya Sotirov Zhechev +2
Semantic caching cuts LLM inference costs by serving a cached response to semantically similar queries. Standard practice evaluates these systems using PR-AUC, a metric that only m…
Training for Compositional Sensitivity Reduces Dense Retrieval Generalization
Radoslav Ralev, Aditeya Baral, Iliya Zhechev +2
Dense retrieval compresses texts into single embeddings ranked by cosine similarity. While efficient for recall, this interface is brittle for identity-level matching: minimal comp…
KGHaluBench: A Knowledge Graph-Based Hallucination Benchmark for Evaluating the Breadth and Depth of LLM Knowledge
Alex Robertson, Huizhi Liang, Mahbub Gani +2
Large Language Models (LLMs) possess a remarkable capacity to generate persuasive and intelligible language. However, coherence does not equate to truthfulness, as the responses of…
Advancing Semantic Caching for LLMs with Domain-Specific Embeddings and Synthetic Data
Waris Gill, Justin Cechmanek, Tyler Hutcherson +5
This report investigates enhancing semantic caching effectiveness by employing specialized, fine-tuned embedding models. Semantic caching relies on embedding similarity rather than…
Ensemble based approach to quantifying uncertainty of LLM based classifications
Srijith Rajamohan, Ahmed Salhin, Josh Frazier +3
The output of Large Language Models (LLMs) are a function of the internal model's parameters and the input provided into the context window. The hypothesis presented here is that u…