137 citations · 410 across the 22 of their papers we have counts for
22 papers
World Models Should Prioritize the Unification of Physical and Social Dynamics
Xiaoyuan Zhang, Chengdong Ma, Yizhe Huang +5
World models, which explicitly learn environmental dynamics to lay the foundation for planning, reasoning, and decision-making, are rapidly advancing in predicting both physical dy…
Differentiable Information Enhanced Model-Based Reinforcement Learning
Xiaoyuan Zhang, Xinyan Cai, Bo Liu +4
Differentiable environments have heralded new possibilities for learning control policies by offering rich differentiable information that facilitates gradient-based methods. In co…
AdaSociety: An Adaptive Environment with Social Structures for Multi-Agent Decision-Making
Yizhe Huang, Xingbo Wang, Hao Liu +7
Traditional interactive environments limit agents' intelligence growth with fixed tasks. Recently, single-agent environments address this by generating new tasks based on agent act…
Learning to Balance Altruism and Self-interest Based on Empathy in Mixed-Motive Games
Fanqi Kong, Yizhe Huang, Song-Chun Zhu +2
Real-world multi-agent scenarios often involve mixed motives, demanding altruistic agents capable of self-protection against potential exploitation. However, existing approaches of…
E^2VPT: An Effective and Efficient Approach for Visual Prompt Tuning
Cheng Han, Qifan Wang, Yiming Cui +4
As the size of transformer-based models continues to grow, fine-tuning these large-scale pretrained vision models for new tasks has become increasingly parameter-intensive. Paramet…
Heterogeneous Value Alignment Evaluation for Large Language Models
Zhaowei Zhang, Ceyao Zhang, Nian Liu +5
The emergent capabilities of Large Language Models (LLMs) have made it crucial to align their values with those of humans. However, current methodologies typically attempt to assig…
Dark, Beyond Deep: A Paradigm Shift to Cognitive AI with Humanlike Common Sense
Yixin Zhu, Tao Gao, Lifeng Fan +9
Recent progress in deep learning is essentially based on a "big data for small tasks" paradigm, under which massive amounts of data are used to train a classifier for a single narr…
Cascaded Human-Object Interaction Recognition
Tianfei Zhou, Wenguan Wang, Siyuan Qi +2
Rapid progress has been witnessed for human-object interaction (HOI) recognition, but most existing models are confined to single-stage reasoning pipelines. Considering the intrins…
Learning Compositional Neural Information Fusion for Human Parsing
Wenguan Wang, Zhijie Zhang, Siyuan Qi +3
This work proposes to combine neural networks with the compositional hierarchy of human bodies for efficient and complete human parsing. We formulate the approach as a neural infor…
PerspectiveNet: 3D Object Detection from a Single RGB Image via Perspective Points
Siyuan Huang, Yixin Chen, Tao Yuan +3
Detecting 3D objects from a single RGB image is intrinsically ambiguous, thus requiring appropriate prior knowledge and intermediate representations as constraints to reduce the un…
Theory-based Causal Transfer: Integrating Instance-level Induction and Abstract-level Structure Learning
Mark Edmonds, Xiaojian Ma, Siyuan Qi +3
Learning transferable knowledge across similar but different settings is a fundamental component of generalized intelligence. In this paper, we approach the transfer learning chall…
Holistic++ Scene Understanding: Single-view 3D Holistic Scene Parsing and Human Pose Estimation with Human-Object Interaction and Physical Commonsense
Yixin Chen, Siyuan Huang, Tao Yuan +3
We propose a new 3D holistic++ scene understanding problem, which jointly tackles two tasks from a single-view image: (i) holistic scene parsing and reconstruction---3D estimations…
Reasoning Visual Dialogs with Structural and Partial Observations
Zilong Zheng, Wenguan Wang, Siyuan Qi +1
We propose a novel model to address the task of Visual Dialog which exhibits complex dialog structures. To obtain a reasonable answer based on the current question and the dialog h…
VRGym: A Virtual Testbed for Physical and Interactive AI
Xu Xie, Hangxin Liu, Zhenliang Zhang +5
We propose VRGym, a virtual reality testbed for realistic human-robot interaction. Different from existing toolkits and virtual reality environments, the VRGym emphasizes on buildi…
Cooperative Holistic Scene Understanding: Unifying 3D Object, Layout, and Camera Pose Estimation
Siyuan Huang, Siyuan Qi, Yinxue Xiao +3
Holistic 3D indoor scene understanding refers to jointly recovering the i) object bounding boxes, ii) room layout, and iii) camera pose, all in 3D. The existing methods either are…
Human-centric Indoor Scene Synthesis Using Stochastic Grammar
Siyuan Qi, Yixin Zhu, Siyuan Huang +2
We present a human-centric method to sample and synthesize 3D room layouts and 2D images thereof, to obtain large-scale 2D/3D image data with perfect per-pixel ground truth. An att…
Learning Human-Object Interactions by Graph Parsing Neural Networks
Siyuan Qi, Wenguan Wang, Baoxiong Jia +2
This paper addresses the task of detecting and recognizing human-object interactions (HOI) in images and videos. We introduce the Graph Parsing Neural Network (GPNN), a framework t…
Holistic 3D Scene Parsing and Reconstruction from a Single RGB Image
Siyuan Huang, Siyuan Qi, Yixin Zhu +3
We propose a computational framework to jointly parse a single RGB image and reconstruct a holistic 3D configuration composed by a set of CAD models using a stochastic grammar mode…
Generalized Earley Parser: Bridging Symbolic Grammars and Sequence Data for Future Prediction
Siyuan Qi, Baoxiong Jia, Song-Chun Zhu
Future predictions on sequence data (e.g., videos or audios) require the algorithms to capture non-Markovian and compositional properties of high-level semantics. Context-free gram…
Intent-aware Multi-agent Reinforcement Learning
Siyuan Qi, Song-Chun Zhu
This paper proposes an intent-aware multi-agent planning framework as well as a learning algorithm. Under this framework, an agent plans in the goal space to maximize the expected…
Predicting Human Activities Using Stochastic Grammar
Siyuan Qi, Siyuan Huang, Ping Wei +1
This paper presents a novel method to predict future human activities from partially observed RGB-D videos. Human activity prediction is generally difficult due to its non-Markovia…
Configurable 3D Scene Synthesis and 2D Image Rendering with Per-Pixel Ground Truth using Stochastic Grammars
Chenfanfu Jiang, Siyuan Qi, Yixin Zhu +5
We propose a systematic learning-based approach to the generation of massive quantities of synthetic 3D scenes and arbitrary numbers of photorealistic 2D images thereof, with assoc…