58 citations · 300 across the 26 of their papers we have counts for
26 papers
The Cultural Psychology of Large Language Models: Is ChatGPT a Holistic or Analytic Thinker?
Chuanyang Jin, Songyang Zhang, Tianmin Shu +1
The prevalent use of Large Language Models (LLMs) has necessitated studying their mental models, yielding noteworthy theoretical and practical implications. Current research has de…
Neural Amortized Inference for Nested Multi-agent Reasoning
Kunal Jha, Tuan Anh Le, Chuanyang Jin +3
Multi-agent interactions, such as communication, teaching, and bluffing, often rely on higher-order social inference, i.e., understanding how others infer oneself. Such intricate r…
Diagnosis, Feedback, Adaptation: A Human-in-the-Loop Framework for Test-Time Policy Adaptation
Andi Peng, Aviv Netanyahu, Mark Ho +4
Policies often fail due to distribution shift -- changes in the state and reward that occur when a policy is deployed in new environments. Data augmentation can increase robustness…
Building Cooperative Embodied Agents Modularly with Large Language Models
Hongxin Zhang, Weihua Du, Jiaming Shan +5
In this work, we address challenging multi-agent cooperation problems with decentralized control, raw sensory observations, costly communication, and multi-objective tasks instanti…
Language Models Meet World Models: Embodied Experiences Enhance Language Models
Jiannan Xiang, Tianhua Tao, Yi Gu +4
While large language models (LMs) have shown remarkable capabilities across numerous tasks, they often struggle with simple reasoning and planning in physical environments, such as…
NOPA: Neurally-guided Online Probabilistic Assistance for Building Socially Intelligent Home Assistants
Xavier Puig, Tianmin Shu, Joshua B. Tenenbaum +1
In this work, we study how to build socially intelligent robots to assist people in their homes. In particular, we focus on assistance with online goal inference, where robots must…
Discovering Generalizable Spatial Goal Representations via Graph-based Active Reward Learning
Aviv Netanyahu, Tianmin Shu, Joshua Tenenbaum +1
In this work, we consider one-shot imitation learning for object rearrangement tasks, where an AI agent needs to watch a single expert demonstration and learn to perform the same t…
Stateful active facilitator: Coordination and Environmental Heterogeneity in Cooperative Multi-Agent Reinforcement Learning
Dianbo Liu, Vedant Shah, Oussama Boussif +6
In cooperative multi-agent reinforcement learning, a team of agents works together to achieve a common goal. Different environments or tasks may require varying degrees of coordina…
Coordinating Policies Among Multiple Agents via an Intelligent Communication Channel
Dianbo Liu, Vedant Shah, Oussama Boussif +6
In Multi-Agent Reinforcement Learning (MARL), specialized channels are often introduced that allow agents to communicate directly with one another. In this paper, we propose an alt…
RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning
Mingkai Deng, Jianyu Wang, Cheng-Ping Hsieh +6
Prompting has shown impressive success in enabling large pretrained language models (LMs) to perform diverse NLP tasks, especially when only few downstream data are available. Auto…
Incorporating Rich Social Interactions Into MDPs
Ravi Tejwani, Yen-Ling Kuo, Tianmin Shu +5
Much of what we do as humans is engage socially with other agents, a skill that robots must also eventually possess. We demonstrate that a rich theory of social interactions origin…
PHASE: PHysically-grounded Abstract Social Events for Machine Social Perception
Aviv Netanyahu, Tianmin Shu, Boris Katz +2
The ability to perceive and reason about social interactions in the context of physical environments is core to human social intelligence and human-machine cooperation. However, no…
Show Me What You Can Do: Capability Calibration on Reachable Workspace for Human-Robot Collaboration
Xiaofeng Gao, Luyao Yuan, Tianmin Shu +2
Aligning humans' assessment of what a robot can do with its true capability is crucial for establishing a common ground between human and robot partners when they collaborate on a…
AGENT: A Benchmark for Core Psychological Reasoning
Tianmin Shu, Abhishek Bhandwaldar, Chuang Gan +6
For machine agents to successfully interact with humans in real-world settings, they will need to develop an understanding of human mental life. Intuitive psychology, the ability t…
Watch-And-Help: A Challenge for Social Perception and Human-AI Collaboration
Xavier Puig, Tianmin Shu, Shuang Li +5
In this paper, we introduce Watch-And-Help (WAH), a challenge for testing social intelligence in agents. In WAH, an AI agent needs to help a human-like agent perform a complex hous…
Active Visual Information Gathering for Vision-Language Navigation
Hanqing Wang, Wenguan Wang, Tianmin Shu +2
Vision-language navigation (VLN) is the task of entailing an agent to carry out navigational instructions inside photo-realistic environments. One of the key challenges in VLN is h…
Joint Mind Modeling for Explanation Generation in Complex Human-Robot Collaborative Tasks
Xiaofeng Gao, Ran Gong, Yizhou Zhao +3
Human collaborators can effectively communicate with their partners to finish a common task by inferring each other's mental states (e.g., goals, beliefs, and desires). Such mind-a…
VRKitchen: an Interactive 3D Virtual Environment for Task-oriented Learning
Xiaofeng Gao, Ran Gong, Tianmin Shu +3
One of the main challenges of advancing task-oriented learning such as visual task planning and reinforcement learning is the lack of realistic and standardized environments for tr…
Interactive Agent Modeling by Learning to Probe
Tianmin Shu, Caiming Xiong, Ying Nian Wu +1
The ability of modeling the other agents, such as understanding their intentions and skills, is essential to an agent's interactions with other agents. Conventional agent modeling…
MRL: Mind-aware Multi-agent Management Reinforcement Learning
Tianmin Shu, Yuandong Tian
Most of the prior work on multi-agent reinforcement learning (MARL) achieves optimal collaboration by directly controlling the agents to maximize a common reward. In this paper, we…
Hierarchical and Interpretable Skill Acquisition in Multi-task Reinforcement Learning
Tianmin Shu, Caiming Xiong, Richard Socher
Learning policies for complex tasks that require multiple different skills is a major challenge in reinforcement learning (RL). It is also a requirement for its deployment in real-…
CERN: Confidence-Energy Recurrent Network for Group Activity Recognition
Tianmin Shu, Sinisa Todorovic, Song-Chun Zhu
This work is about recognizing human activities occurring in videos at distinct semantic levels, including individual actions, interactions, and group activities. The recognition i…
Learning Social Affordance Grammar from Videos: Transferring Human Interactions to Human-Robot Interactions
Tianmin Shu, Xiaofeng Gao, Michael S. Ryoo +1
In this paper, we present a general framework for learning social affordance grammar as a spatiotemporal AND-OR graph (ST-AOG) from RGB-D videos of human interactions, and transfer…
Modeling and Inferring Human Intents and Latent Functional Objects for Trajectory Prediction
Dan Xie, Tianmin Shu, Sinisa Todorovic +1
This paper is about detecting functional objects and inferring human intentions in surveillance videos of public spaces. People in the videos are expected to intentionally take sho…