143 citations · 440 across the 42 of their papers we have counts for
17 papers · 1 filter
What Makes a Maze Look Like a Maze?
Joy Hsu, Jiayuan Mao, Joshua B. Tenenbaum +2
A unique aspect of human visual understanding is the ability to flexibly interpret abstract concepts: acquiring lifted rules explaining what they symbolize, grounding them across f…
What's Left? Concept Grounding with Logic-Enhanced Foundation Models
Joy Hsu, Jiayuan Mao, Joshua B. Tenenbaum +1
Recent works such as VisProg and ViperGPT have smartly composed foundation models for visual reasoning-using large language models (LLMs) to produce programs that can be executed b…
NS3D: Neuro-Symbolic Grounding of 3D Objects and Relations
Joy Hsu, Jiayuan Mao, Jiajun Wu
Grounding object properties and relations in 3D scenes is a prerequisite for a wide range of artificial intelligence tasks, such as visually grounded dialogues and embodied manipul…
Translating a Visual LEGO Manual to a Machine-Executable Plan
Ruocheng Wang, Yunzhi Zhang, Jiayuan Mao +2
We study the problem of translating an image-based, step-by-step assembly manual created by human designers into machine-interpretable instructions. We formulate this problem as a…
Programmatic Concept Learning for Human Motion Description and Synthesis
Sumith Kulal, Jiayuan Mao, Alex Aiken +1
We introduce Programmatic Motion Concepts, a hierarchical motion representation for human actions that captures both low-level motion and high-level description as motion concepts.…
FALCON: Fast Visual Concept Learning by Integrating Images, Linguistic descriptions, and Conceptual Relations
Lingjie Mei, Jiayuan Mao, Ziqi Wang +2
We present a meta-learning framework for learning new visual concepts quickly, from just one or a few examples, guided by multiple naturally occurring data streams: simultaneously…
Hierarchical Motion Understanding via Motion Programs
Sumith Kulal, Jiayuan Mao, Alex Aiken +1
Current approaches to video analysis of human motion focus on raw pixels or keypoints as the basic units of reasoning. We posit that adding higher-level motion primitives, which ca…
Grounding Physical Concepts of Objects and Events Through Dynamic Visual Reasoning
Zhenfang Chen, Jiayuan Mao, Jiajun Wu +3
We study the problem of dynamic visual reasoning on raw videos. This is a challenging problem; currently, state-of-the-art models often require dense supervision on physical object…
Object-Centric Diagnosis of Visual Reasoning
Jianwei Yang, Jiayuan Mao, Jiajun Wu +4
When answering questions about an image, it not only needs knowing what -- understanding the fine-grained contents (e.g., objects, relationships) in the image, but also telling why…
Multi-Plane Program Induction with 3D Box Priors
Yikai Li, Jiayuan Mao, Xiuming Zhang +4
We consider two important aspects in understanding and editing images: modeling regular, program-like texture or patterns in 2D planes, and 3D posing of these planes in the scene.…
Perspective Plane Program Induction from a Single Image
Yikai Li, Jiayuan Mao, Xiuming Zhang +3
We study the inverse graphics problem of inferring a holistic representation for natural images. Given an input image, our goal is to induce a neuro-symbolic, program-like represen…
Visual Concept-Metaconcept Learning
Chi Han, Jiayuan Mao, Chuang Gan +2
Humans reason with concepts and metaconcepts: we recognize red and green from visual input; we also understand that they describe the same property of objects (i.e., the color). In…
Program-Guided Image Manipulators
Jiayuan Mao, Xiuming Zhang, Yikai Li +3
Humans are capable of building holistic representations for images at various levels, from local objects, to pairwise relations, to global structures. The interpretation of structu…
UniVSE: Robust Visual Semantic Embeddings via Structured Semantic Representations
Hao Wu, Jiayuan Mao, Yufeng Zhang +4
We propose Unified Visual-Semantic Embeddings (UniVSE) for learning a joint space of visual and textual concepts. The space unifies the concepts at different levels, including obje…
The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision
Jiayuan Mao, Chuang Gan, Pushmeet Kohli +2
We propose the Neuro-Symbolic Concept Learner (NS-CL), a model that learns visual concepts, words, and semantic parsing of sentences without explicit supervision on any of them; in…
Acquisition of Localization Confidence for Accurate Object Detection
Borui Jiang, Ruixuan Luo, Jiayuan Mao +2
Modern CNN-based object detectors rely on bounding box regression and non-maximum suppression to localize objects. While the probabilities for class labels naturally reflect classi…
What Can Help Pedestrian Detection?
Jiayuan Mao, Tete Xiao, Yuning Jiang +1
Aggregating extra features has been considered as an effective approach to boost traditional pedestrian detection methods. However, there is still a lack of studies on whether and…