13 papers
MindZero: Learning Online Mental Reasoning With Zero Annotations
Shunchi Zhang, Jin Lu, Chuanyang Jin +3
Effective real-world assistance requires AI agents with robust Theory of Mind (ToM): inferring human mental states from their behavior. Despite recent advances, several key challen…
ThoughtTrace: Understanding User Thoughts in Real-World LLM Interactions
Chuanyang Jin, Binze Li, Haopeng Xie +6
Conversational AI has now reached billions of users, yet existing datasets capture only what people say, not what they think. We introduce ThoughtTrace, the first large-scale datas…
AutoToM: Scaling Model-based Mental Inference via Automated Agent Modeling
Zhining Zhang, Chuanyang Jin, Mung Yao Jia +2
Theory of Mind (ToM), the ability to understand people's minds based on their behavior, is key to developing socially intelligent agents. Current approaches to ToM reasoning either…
RealWebAssist: A Benchmark for Long-Horizon Web Assistance with Real-World Users
Suyu Ye, Haojun Shi, Darren Shih +3
To achieve successful assistance with long-horizon web-based tasks, AI agents must be able to sequentially follow real-world user instructions over a long period. Unlike existing w…
Pragmatic Embodied Spoken Instruction Following in Human-Robot Collaboration with Theory of Mind
Lance Ying, Xinyi Li, Shivam Aarya +6
Spoken language instructions are ubiquitous in agent collaboration. However, in real-world human-robot collaboration, following human spoken instructions can be challenging due to…
The Era of Real-World Human Interaction: RL from User Conversations
Chuanyang Jin, Jing Xu, Bo Liu +6
We posit that to achieve continual model improvement and multifaceted alignment, future models must learn from natural human interaction. Current conversational models are aligned…