5 citations · 7 across the 13 of their papers we have counts for
16 papers
GameBoyWorlds: A Testbed for Self-Improvement in Embodied Video Games
Dhananjay Ashok, Adam Shen, Aslan Huo Feng +10
Powered by expert guidance, agents can operate in interactive environments; however, it is unclear whether they can learn autonomously from their own experience. To evaluate such s…
Toward Interactive Understanding of Code APIs
Dhananjay Ashok, Jesse Thomason, Jonathan May
Empowered by advances in Language Model agents, systems have made substantial strides in code generation and understanding. However, these approaches often rely on read access to t…
Vulnerable Code Search: Transferable Attack for Code Language Models
Kaicheng Wang, Liyan Huang, Jesse Thomason +1
Reliable code retrieval is crucial for developer productivity and effective code reuse. However, current neural code language models (CLMs) powering search tools are susceptible to…
Algorithmic Primitives of Theory of Mind Benchmarks
Wang Bill Zhu, Qiutong Tony Yi, Robin Jia +1
Do Theory of Mind (ToM) benchmarks really test theory of mind, or are they actually testing parsing and algorithmic state tracking? While benchmarks vary modality and dialogue leng…
PSALM-V: Automating Symbolic Planning in Interactive Visual Environments with Large Language Models
Wang Bill Zhu, Miaosen Chai, Ishika Singh +2
We propose PSALM-V, the first autonomous neuro-symbolic learning system able to induce symbolic action semantics (i.e., pre- and post-conditions) in visual environments through int…
Why Do Some Inputs Break Low-Bit LLM Quantization?
Ting-Yun Chang, Muru Zhang, Jesse Thomason +1
Low-bit weight-only quantization significantly reduces the memory footprint of large language models (LLMs), but disproportionately affects certain examples. We analyze diverse 3-4…
Large Language Models Do Multi-Label Classification Differently
Marcus Ma, Georgios Chochlakis, Niyantha Maruthu Pandiyan +2
Multi-label classification is prevalent in real-world settings, but the behavior of Large Language Models (LLMs) in this setting is understudied. We investigate how autoregressive…
Iterative Formalization and Planning in Partially Observable Environments
Liancheng Gong, Wang Zhu, Jesse Thomason +1
Using LLMs not to predict plans but to formalize an environment into the Planning Domain Definition Language (PDDL) has been shown to improve performance and control. While most ex…
Evaluating Creativity and Deception in Large Language Models: A Simulation Framework for Multi-Agent Balderdash
Parsa Hejabi, Elnaz Rahmati, Alireza S. Ziabari +3
Large Language Models (LLMs) have shown impressive capabilities in complex tasks and interactive environments, yet their creativity remains underexplored. This paper introduces a s…
When Parts Are Greater Than Sums: Individual LLM Components Can Outperform Full Models
Ting-Yun Chang, Jesse Thomason, Robin Jia
This paper studies in-context learning by decomposing the output of large language models into the individual contributions of attention heads and MLPs (components). We observe cur…
Language Models can Infer Action Semantics for Symbolic Planners from Environment Feedback
Wang Zhu, Ishika Singh, Robin Jia +1
Symbolic planners can discover a sequence of actions from initial to goal states given expert-defined, domain-specific logical action semantics. Large Language Models (LLMs) can di…
TwoStep: Multi-agent Task Planning using Classical Planners and Large Language Models
David Bai, Ishika Singh, David Traum +1
Classical planning formulations like the Planning Domain Definition Language (PDDL) admit action sequences guaranteed to achieve a goal state given an initial state if any are poss…
THE COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic Manipulation
Wilbert Pumacay, Ishika Singh, Jiafei Duan +3
To realize effective large-scale, real-world robotic applications, we must evaluate how well our robot policies adapt to changes in environmental conditions. Unfortunately, a major…
Does VLN Pretraining Work with Nonsensical or Irrelevant Instructions?
Wang Zhu, Ishika Singh, Yuan Huang +2
Data augmentation via back-translation is common when pretraining Vision-and-Language Navigation (VLN) models, even though the generated instructions are noisy. But: does that nois…
Efficient End-to-End Visual Document Understanding with Rationale Distillation
Wang Zhu, Alekh Agarwal, Mandar Joshi +3
Understanding visually situated language requires interpreting complex layouts of textual and visual elements. Pre-processing tools, such as optical character recognition (OCR), ca…
Do Localization Methods Actually Localize Memorized Data in LLMs? A Tale of Two Benchmarks
Ting-Yun Chang, Jesse Thomason, Robin Jia
The concept of localization in LLMs is often mentioned in prior work; however, methods for localization have never been systematically and directly evaluated. We propose two comple…