47 citations · 55 across the 10 of their papers we have counts for
10 papers
Autonomous Assessment of Generalizability of AI Agent Capabilities
Daniel Bramblett, Rushang Karia, Adrian Ciotinga +3
Safe deployment of black-box AI (BBAI) systems such as foundation model agents requires methods for evaluating their capabilities in novel settings. We define an agent's capability…
Autonomous Evaluation of LLMs for Truth Maintenance and Reasoning Tasks
Rushang Karia, Daniel Bramblett, Daksh Dobhal +1
This paper presents AutoEval, a novel benchmark for scaling Large Language Model (LLM) assessment in formal tasks with clear notions of correctness, such as truth maintenance in tr…
Using Explainable AI and Hierarchical Planning for Outreach with Robots
Rushang Karia, Jayesh Nagpal, Daksh Dobhal +4
Understanding how robots plan and execute tasks is crucial in today's world, where they are becoming more prevalent in our daily lives. However, teaching non-experts, such as K-12…
utoval: Autonomous Assessment of LLMs in Formal Synthesis and Interpretation Tasks
Rushang Karia, Daniel Bramblett, Daksh Dobhal +2
This paper presents utoval, a new approach for scaling LLM assessment in translating formal syntax -- such as first-order logic, regular expressions, etc -- to na…
Epistemic Exploration for Generalizable Planning and Learning in Non-Stationary Settings
Rushang Karia, Pulkit Verma, Alberto Speranzon +1
This paper introduces a new approach for continual planning and model learning in relational, non-stationary stochastic environments. Such capabilities are essential for the deploy…
Autonomous Capability Assessment of Sequential Decision-Making Systems in Stochastic Settings (Extended Version)
Pulkit Verma, Rushang Karia, Siddharth Srivastava
It is essential for users to understand what their AI systems can and can't do in order to use them safely. However, the problem of enabling users to assess AI systems with sequent…
Relational Abstractions for Generalized Reinforcement Learning on Symbolic Problems
Rushang Karia, Siddharth Srivastava
Reinforcement learning in problems with symbolic state spaces is challenging due to the need for reasoning over long horizons. This paper presents a new approach that utilizes rela…
Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks
Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi +37
How well can NLP models generalize to a variety of unseen tasks when provided with task instructions? To address this question, we first introduce Super-NaturalInstructions, a benc…
Learning Generalized Policy Automata for Relational Stochastic Shortest Path Problems
Rushang Karia, Rashmeet Kaur Nayyar, Siddharth Srivastava
Several goal-oriented problems in the real-world can be naturally expressed as Stochastic Shortest Path Problems (SSPs). However, the computational complexity of solving SSPs makes…
Learning Generalized Relational Heuristic Networks for Model-Agnostic Planning
Rushang Karia, Siddharth Srivastava
Computing goal-directed behavior is essential to designing efficient AI systems. Due to the computational complexity of planning, current approaches rely primarily upon hand-coded…