11 citations · 22 across the 11 of their papers we have counts for
12 papers
SpeedrunBench: Challenging LLM Agents with Video Game Speedrunning
Yoshinari Fujinuma, Keisuke Kamahori, Ryuto Koike +8
Frontier LLM agents have been shown to be capable of solving increasingly complex tasks for which humans have measurable solutions. This begs the pertinent question of whether LLM…
FigmaTrace: Capturing Creative Nuances in Human Figma Design Workflows
Darshan Deshpande, Yoshinari Fujinuma, Martyna Markiewicz +5
Vision Language Models have recently shown improvements in several objective and verifiable domains such as object detection but continue to underperform on subjective and creative…
Defenses & Enablers For Skill Injection Attacks on Terminal Based Agents
Yoshinari Fujinuma, Varun Gangal, Traian Rebedea +4
Large language model (LLM) agents increasingly rely on reusable skills i.e. documents describing task-specific procedures. However, this introduces a new attack surface for agents…
DETOUR: An Interactive Benchmark for Dual-Agent Search and Reasoning
Li Siyan, Darshan Deshpande, Anand Kannappan +1
When recalling information in conversation, people often arrive at the recollection after multiple turns. However, existing benchmarks for evaluating agent capabilities in such tip…
Benchmarking Reward Hack Detection in Code Environments via Contrastive Analysis
Darshan Deshpande, Anand Kannappan, Rebecca Qian
Recent advances in reinforcement learning for code generation have made robust environments essential to prevent reward hacking. As LLMs increasingly serve as evaluators in code-ba…
MEMTRACK: Evaluating Long-Term Memory and State Tracking in Multi-Platform Dynamic Agent Environments
Darshan Deshpande, Varun Gangal, Hersh Mehta +3
Recent works on context and memory benchmarking have primarily focused on conversational instances but the need for evaluating memory in dynamic enterprise environments is crucial…
TRAIL: Trace Reasoning and Agentic Issue Localization
Darshan Deshpande, Varun Gangal, Hersh Mehta +3
The increasing adoption of agentic workflows across diverse domains brings a critical need to scalably and systematically evaluate the complex traces these systems generate. Curren…
Browsing Lost Unformed Recollections: A Benchmark for Tip-of-the-Tongue Search and Reasoning
Sky CH-Wang, Darshan Deshpande, Smaranda Muresan +2
We introduce Browsing Lost Unformed Recollections, a tip-of-the-tongue known-item search and reasoning benchmark for general AI assistants. BLUR introduces a set of 573 real-world…
GLIDER: Grading LLM Interactions and Decisions using Explainable Ranking
Darshan Deshpande, Selvan Sunitha Ravi, Sky CH-Wang +3
The LLM-as-judge paradigm is increasingly being adopted for automated evaluation of model outputs. While LLM judges have shown promise on constrained evaluation tasks, closed sourc…
Lynx: An Open Source Hallucination Evaluation Model
Selvan Sunitha Ravi, Bartosz Mielczarek, Anand Kannappan +2
Retrieval Augmented Generation (RAG) techniques aim to mitigate hallucinations in Large Language Models (LLMs). However, LLMs can still produce information that is unsupported or c…
FinanceBench: A New Benchmark for Financial Question Answering
Pranab Islam, Anand Kannappan, Douwe Kiela +3
FinanceBench is a first-of-its-kind test suite for evaluating the performance of LLMs on open book financial question answering (QA). It comprises 10,231 questions about publicly t…
SimpleSafetyTests: a Test Suite for Identifying Critical Safety Risks in Large Language Models
Bertie Vidgen, Nino Scherrer, Hannah Rose Kirk +4
The past year has seen rapid acceleration in the development of large language models (LLMs). However, without proper steering and safeguards, LLMs will readily follow malicious in…