2 citations · 5 across the 9 of their papers we have counts for
7 papers · 1 filter
Articulation Upper Bounds for Language Models
Alexander Spangher, Daniel E. Ho, Diyi Yang +2
Language captures only a subset of the perceptions, intuitions and intents that form meaning between speakers. What, then, are the limits of language? We take steps towards measuri…
Verifiable, Articulable, and Tacit Components of Preference
Alexander Spangher, Sheldon S. Huang, Sheldon Huang +5
What makes a short story gripping; a news article newsworthy; or a math proof elegant? These constructs resist articulation or verification; their meaning is at least partially tac…
MobileCybench: Evaluating Agent Vulnerability Discovery via Executable Probes
Andy K. Zhang, Ava Huang, Joey Ji +21
AI agents now report vulnerabilities faster than maintainers can review them. Reports often depend on security properties specific to the application, and require considerable huma…
API Benchmark Scores Do Not Reliably Transfer to Chatbot Interfaces
Jennifer Wang, Joachim Baumann, Daniel E. Ho +1
Benchmark scores are a central currency in model releases: they inform purchasing decisions, shape public trust, and influence policy. Yet, a key assumption underlying benchmark sc…
What AI Benchmarks Actually Measure: Adapting Convergent and Discriminant Validity to Interrogate Fifty-Six AI Benchmarks
Meera Desai, Sang T. Truong, Hanna Wallach +8
Benchmarks play a central role in the development and governance of models, yet it is often unclear whether they actually measure the concepts they purport to measure (e.g., reason…
Extractable Memorization From First Principles
A. Feder Cooper, Marika Swanberg, Jamie Hayes +6
Recent work on extractable memorization in language models suffers from two contrasting validity problems. Some studies overstate extraction, for example, by using sequences too sh…
Estimating near-verbatim extraction risk in language models with decoding-constrained beam search
A. Feder Cooper, Mark A. Lemley, Christopher De Sa +6
Recent work shows that standard greedy-decoding extraction methods for quantifying memorization in LLMs miss how extraction risk varies across sequences. Probabilistic extraction -…