29 citations · 108 across the 19 of their papers we have counts for
20 papers
MemoBench: Benchmarking World Modeling in Dynamically Changing Environments
Haoyu Chen, Kaichen Zhou, Hang Hua +11
Video generation models aspire to simulate dynamic environments, and several benchmarks now evaluate memory consistency across frames. However, most assess consistency only while t…
SimWorlds: A Multi-Agent System for Dynamic 3D Scene Creation
Chunjiang Liu, Xiaoyuan Wang, Haoyu Chen +3
LLM agents are increasingly used to translate natural language into 3D scenes in a procedural way, but existing systems focus on static output. Dynamic 4D scenes from text alone, i…
Sim2Plan: Robot Motion Planning via Message Passing between Simulation and Reality
Yizhou Zhao, Yuanhong Zeng, Qian Long +2
Simulation-to-real is the task of training and developing machine learning models and deploying them in real settings with minimal additional training. This approach is becoming in…
ARNOLD: A Benchmark for Language-Grounded Task Learning With Continuous States in Realistic 3D Scenes
Ran Gong, Jiangyong Huang, Yizhou Zhao +9
Understanding the continuous states of objects is essential for task learning and planning in the real world. However, most existing task learning benchmarks assume discrete (e.g.,…
OpenD: A Benchmark for Language-Driven Door and Drawer Opening
Yizhou Zhao, Qiaozi Gao, Liang Qiu +2
We introduce OPEND, a benchmark for learning how to use a hand to open cabinet doors or drawers in a photo-realistic and physics-reliable simulation environment driven by language…
VRKitchen2.0-IndoorKit: A Tutorial for Augmented Indoor Scene Building in Omniverse
Yizhou Zhao, Steven Gong, Xiaofeng Gao +2
With the recent progress of simulations by 3D modeling software and game engines, many researchers have focused on Embodied AI tasks in the virtual environment. However, the resear…
Triangular Character Animation Sampling with Motion, Emotion, and Relation
Yizhou Zhao, Liang Qiu, Wensi Ai +2
Dramatic progress has been made in animating individual characters. However, we still lack automatic control over activities between characters, especially those involving interact…
Learning to Act with Affordance-Aware Multimodal Neural SLAM
Zhiwei Jia, Kaixiang Lin, Yizhou Zhao +3
Recent years have witnessed an emerging paradigm shift toward embodied artificial intelligence, in which an agent must learn to solve challenging tasks by interacting with its envi…
ValueNet: A New Dataset for Human Value Driven Dialogue System
Liang Qiu, Yizhou Zhao, Jinchao Li +4
Building a socially intelligent agent involves many challenges, one of which is to teach the agent to speak guided by its value like a human. However, value-driven chatbots are sti…
Learning from the Tangram to Solve Mini Visual Tasks
Yizhou Zhao, Liang Qiu, Pan Lu +3
Current pre-training methods in computer vision focus on natural images in the daily-life context. However, abstract diagrams such as icons and symbols are common and important in…
GenMotion: Data-driven Motion Generators for Real-time Animation Synthesis
Yizhou Zhao, Wensi Ai, Liang Qiu +4
With the recent success of deep learning algorithms, many researchers have focused on generative models for human motion animation. However, the research community lacks a platform…
LUMINOUS: Indoor Scene Generation for Embodied AI Challenges
Yizhou Zhao, Kaixiang Lin, Zhiwei Jia +4
Learning-based methods for training embodied agents typically require a large number of high-quality scenes that contain realistic layouts and support meaningful interactions. Howe…
IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoning
Pan Lu, Liang Qiu, Jiaqi Chen +6
Current visual question answering (VQA) tasks mainly consider answering human-annotated questions for natural images. However, aside from natural images, abstract diagrams with sem…
STAR: Sparse Transformer-based Action Recognition
Feng Shi, Chonghan Lee, Liang Qiu +6
The cognitive system for human action and behavior has evolved into a deep learning regime, and especially the advent of Graph Convolution Networks has transformed the field in rec…
SocAoG: Incremental Graph Parsing for Social Relation Inference in Dialogues
Liang Qiu, Yuan Liang, Yizhou Zhao +5
Inferring social relations from dialogues is vital for building emotionally intelligent robots to interpret human language better and act accordingly. We model the social network a…
Towards Socially Intelligent Agents with Mental State Transition and Human Utility
Liang Qiu, Yizhou Zhao, Yuan Liang +4
Building a socially intelligent agent involves many challenges. One of which is to track the agent's mental state transition and teach the agent to make decisions guided by its val…
Weighted Entropy Modification for Soft Actor-Critic
Yizhou Zhao, Song-Chun Zhu
We generalize the existing principle of the maximum Shannon entropy in reinforcement learning (RL) to weighted entropy by characterizing the state-action pairs with some qualitativ…
Vertical-Horizontal Structured Attention for Generating Music with Chords
Yizhou Zhao, Liang Qiu, Wensi Ai +2
In this paper, we propose a lightweight music-generating model based on variational autoencoder (VAE) with structured attention. Generating music is different from generating text…
Structured Attention for Unsupervised Dialogue Structure Induction
Liang Qiu, Yizhou Zhao, Weiyan Shi +5
Inducing a meaningful structural representation from one or a set of dialogues is a crucial but challenging task in computational linguistics. Advancement made in this area is crit…
Joint Mind Modeling for Explanation Generation in Complex Human-Robot Collaborative Tasks
Xiaofeng Gao, Ran Gong, Yizhou Zhao +3
Human collaborators can effectively communicate with their partners to finish a common task by inferring each other's mental states (e.g., goals, beliefs, and desires). Such mind-a…