7 papers
Learning the ARTS of Search for Automated Discovery
Gurusha Juneja, Arnav Kumar Jain, Deepak Nathani +2
Scientific discovery can be formulated as an iterative search process over the space of hypotheses and experiments. Contemporary methods navigate this space using heuristics such a…
EnactToM: An Evolving Benchmark for Functional Theory of Mind in Embodied Agents
Gurusha Juneja, Dylan Lu, Saaket Agashe +7
Theory of Mind (ToM), the ability to track others epistemic state, makes humans efficient collaborators. AI agents need the same capacity in multi agent settings, yet existing benc…
Proactive Agent Research Environment: Simulating Active Users to Evaluate Proactive Assistants
Deepak Nathani, Cheng Zhang, Chang Huan +7
Proactive agents that anticipate user needs and autonomously execute tasks hold great promise as digital assistants, yet the lack of realistic user simulation frameworks hinders th…
Do You Know About My Nation? Investigating Multilingual Language Models' Cultural Literacy Through Factual Knowledge
Eshaan Tanwar, Anwoy Chatterjee, Michael Saxon +3
Most multilingual question-answering benchmarks, while covering a diverse pool of languages, do not factor in regional diversity in the information they capture and tend to be West…
Benchmarks as Microscopes: A Call for Model Metrology
Michael Saxon, Ari Holtzman, Peter West +2
Modern language models (LMs) pose a new challenge in capability assessment. Static benchmarks inevitably saturate without providing confidence in the deployment tolerances of LM-ba…
VSP: Assessing the dual challenges of perception and reasoning in spatial planning tasks for VLMs
Qiucheng Wu, Handong Zhao, Michael Saxon +4
Vision language models (VLMs) are an exciting emerging class of language models (LMs) that have merged classic LM capabilities with those of image processing systems. However, the…