11 citations · 22 across the 11 of their papers we have counts for
5 papers · 1 filter
SpeedrunBench: Challenging LLM Agents with Video Game Speedrunning
Yoshinari Fujinuma, Keisuke Kamahori, Ryuto Koike +8
Frontier LLM agents have been shown to be capable of solving increasingly complex tasks for which humans have measurable solutions. This begs the pertinent question of whether LLM…
MEMTRACK: Evaluating Long-Term Memory and State Tracking in Multi-Platform Dynamic Agent Environments
Darshan Deshpande, Varun Gangal, Hersh Mehta +3
Recent works on context and memory benchmarking have primarily focused on conversational instances but the need for evaluating memory in dynamic enterprise environments is crucial…
TRAIL: Trace Reasoning and Agentic Issue Localization
Darshan Deshpande, Varun Gangal, Hersh Mehta +3
The increasing adoption of agentic workflows across diverse domains brings a critical need to scalably and systematically evaluate the complex traces these systems generate. Curren…
Browsing Lost Unformed Recollections: A Benchmark for Tip-of-the-Tongue Search and Reasoning
Sky CH-Wang, Darshan Deshpande, Smaranda Muresan +2
We introduce Browsing Lost Unformed Recollections, a tip-of-the-tongue known-item search and reasoning benchmark for general AI assistants. BLUR introduces a set of 573 real-world…
Lynx: An Open Source Hallucination Evaluation Model
Selvan Sunitha Ravi, Bartosz Mielczarek, Anand Kannappan +2
Retrieval Augmented Generation (RAG) techniques aim to mitigate hallucinations in Large Language Models (LLMs). However, LLMs can still produce information that is unsupported or c…