177 citations · 351 across the 22 of their papers we have counts for
4 papers · 1 filter
SimBA: Simplifying Benchmark Analysis Using Performance Matrices Alone
Nishant Subramani, Alfredo Gomez, Mona Diab
Modern language models are evaluated on large benchmarks, which are difficult to make sense of, especially for model selection. Looking at the raw evaluation numbers themselves usi…
LLM Microscope: What Model Internals Reveal About Answer Correctness and Context Utilization
Jiarui Liu, Jivitesh Jain, Mona Diab +1
Although large language models (LLMs) have tremendous utility, trustworthiness is still a chief concern: models often generate incorrect information with high confidence. While con…
Model Internal Sleuthing: Finding Lexical Identity and Inflectional Features in Modern Language Models
Michael Li, Nishant Subramani
Large transformer-based language models dominate modern NLP, yet our understanding of how they encode linguistic information relies primarily on studies of early models like BERT a…
MICE for CATs: Model-Internal Confidence Estimation for Calibrating Agents with Tools
Nishant Subramani, Jason Eisner, Justin Svegliato +3
Tool-using agents that act in the world need to be both useful and safe. Well-calibrated model confidences can be used to weigh the risk versus reward of potential actions, but pri…