6 papers
Evaluation Awareness Is Not One Capability: Evidence from Open Language Models
Nilesh Nayan, Aishwarya Sampath Kumar, Rishiraj Girmal +5
Safety benchmarks assume that test-condition behavior predicts deployment behavior, an assumption that fails if models detect evaluation cues and adapt. This opens a gap between be…
Hop, Skip, and Overthink: Diagnosing Why Reasoning Models Fumble during Multi-Hop Analysis
Anushka Yadav, Isha Nalawade, Srujana Pillarichety +7
The emergence of reasoning models and their integration into practical AI chat bots has led to breakthroughs in solving advanced math, deep search, and extractive question answerin…
Quantifying reliance on external information over parametric knowledge during Retrieval Augmented Generation (RAG) using mechanistic analysis
Reshmi Ghosh, Rahul Seetharaman, Hitesh Wadhwa +6
Retrieval Augmented Generation (RAG) is a widely used approach for leveraging external context in several natural language applications such as question answering and information r…
From RAGs to rich parameters: Probing how language models utilize external knowledge over parametric information for factual queries
Hitesh Wadhwa, Rahul Seetharaman, Somyaa Aggarwal +6
Retrieval Augmented Generation (RAG) enriches the ability of language models to reason using external context to augment responses for a given user prompt. This approach has risen…
Topic Segmentation of Semi-Structured and Unstructured Conversational Datasets using Language Models
Reshmi Ghosh, Harjeet Singh Kajal, Sharanya Kamath +4
Breaking down a document or a conversation into multiple contiguous segments based on its semantic structure is an important and challenging problem in NLP, which can assist many d…
On Surgical Fine-tuning for Language Encoders
Abhilasha Lodha, Gayatri Belapurkar, Saloni Chalkapurkar +5
Fine-tuning all the layers of a pre-trained neural language encoder (either using all the parameters or using parameter-efficient methods) is often the de-facto way of adapting it…