1.6k citations · 9.7k across the 222 of their papers we have counts for
34 papers · 1 filter
Object-Centric Diagnosis of Visual Reasoning
Jianwei Yang, Jiayuan Mao, Jiajun Wu +4
When answering questions about an image, it not only needs knowing what -- understanding the fine-grained contents (e.g., objects, relationships) in the image, but also telling why…
Representing Partial Programs with Blended Abstract Semantics
Maxwell Nye, Yewen Pu, Matthew Bowers +3
Synthesizing programs from examples requires searching over a vast, combinatorial space of possible programs. In this search process, a key challenge is representing the behavior o…
Augmenting Policy Learning with Routines Discovered from a Single Demonstration
Zelin Zhao, Chuang Gan, Jiajun Wu +2
Humans can abstract prior knowledge from very little data and use it to boost skill learning. In this paper, we propose routine-augmented policy learning (RAPL), which discovers ro…
Neural Radiance Flow for 4D View Synthesis and Video Processing
Yilun Du, Yinan Zhang, Hong-Xing Yu +2
We present a method, Neural Radiance Flow (NeRFlow),to learn a 4D spatial-temporal representation of a dynamic scene from a set of RGB images. Key to our approach is the use of a n…
Improved Contrastive Divergence Training of Energy Based Models
Yilun Du, Shuang Li, Joshua Tenenbaum +1
Contrastive divergence is a popular method of training energy-based models, but is known to have difficulties with training stability. We propose an adaptation to improve contrasti…
Multi-Plane Program Induction with 3D Box Priors
Yikai Li, Jiayuan Mao, Xiuming Zhang +4
We consider two important aspects in understanding and editing images: modeling regular, program-like texture or patterns in 2D planes, and 3D posing of these planes in the scene.…
A Long Horizon Planning Framework for Manipulating Rigid Pointcloud Objects
Anthony Simeonov, Yilun Du, Beomjoon Kim +4
We present a framework for solving long-horizon planning problems involving manipulation of rigid objects that operates directly from a point-cloud observation, i.e. without prior…
Building 3D Morphable Models from a Single Scan
Skylar Sutherland, Bernhard Egger, Joshua Tenenbaum
We propose a method for constructing generative models of 3D objects from a single 3D mesh. Our method produces a 3D morphable model that represents shape and albedo in terms of Ga…
Data-Efficient Learning for Complex and Real-Time Physical Problem Solving using Augmented Simulation
Kei Ota, Devesh K. Jha, Diego Romeres +7
Humans quickly solve tasks in novel systems with complex dynamics, without requiring much interaction. While deep reinforcement learning algorithms have achieved tremendous success…
Improving the Reconstruction of Disentangled Representation Learners via Multi-Stage Modeling
Akash Srivastava, Yamini Bansal, Yukun Ding +15
Current autoencoder-based disentangled representation learning methods achieve disentanglement by penalizing the (aggregate) posterior to encourage statistical independence of the…
Watch-And-Help: A Challenge for Social Perception and Human-AI Collaboration
Xavier Puig, Tianmin Shu, Shuang Li +5
In this paper, we introduce Watch-And-Help (WAH), a challenge for testing social intelligence in agents. In WAH, an AI agent needs to help a human-like agent perform a complex hous…
Planning with Learned Object Importance in Large Problem Instances using Graph Neural Networks
Tom Silver, Rohan Chitnis, Aidan Curtis +3
Real-world planning problems often involve hundreds or even thousands of objects, straining the limits of modern planners. In this work, we address this challenge by learning to pr…
Learning abstract structure for drawing by efficient motor program induction
Lucas Y. Tian, Kevin Ellis, Marta Kryven +1
Humans flexibly solve new problems that differ qualitatively from those they were trained on. This ability to generalize is supported by learned concepts that capture structure com…
Noisy Agents: Self-supervised Exploration by Predicting Auditory Events
Chuang Gan, Xiaoyu Chen, Phillip Isola +2
Humans integrate multiple sensory modalities (e.g. visual and audio) to build a causal understanding of the physical world. In this work, we propose a novel type of intrinsic motiv…
End-to-End Optimization of Scene Layout
Andrew Luo, Zhoutong Zhang, Jiajun Wu +1
We propose an end-to-end variational generative model for scene layout synthesis conditioned on scene graphs. Unlike unconditional scene layout generation, we use scene graphs as a…
Learning to learn generative programs with Memoised Wake-Sleep
Luke B. Hewitt, Tuan Anh Le, Joshua B. Tenenbaum
We study a class of neuro-symbolic generative models in which neural networks are used both for inference and as priors over symbolic, data-generating programs. As generative model…
Foley Music: Learning to Generate Music from Videos
Chuang Gan, Deng Huang, Peihao Chen +2
In this paper, we introduce Foley Music, a system that can synthesize plausible music for a silent video clip about people playing musical instruments. We first identify two key in…
Unsupervised Discovery of 3D Physical Objects from Video
Yilun Du, Kevin Smith, Tomer Ulman +2
We study the problem of unsupervised physical object discovery. While existing frameworks aim to decompose scenes into 2D segments based off each object's appearance, we explore ho…
Program Synthesis with Pragmatic Communication
Yewen Pu, Kevin Ellis, Marta Kryven +2
Program synthesis techniques construct or infer programs from user-provided specifications, such as input-output examples. Yet most specifications, especially those given by end-us…
ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulation
Chuang Gan, Jeremy Schwartz, Seth Alter +21
We introduce ThreeDWorld (TDW), a platform for interactive multi-modal physical simulation. TDW enables simulation of high-fidelity sensory data and physical interactions between m…
Perspective Plane Program Induction from a Single Image
Yikai Li, Jiayuan Mao, Xiuming Zhang +3
We study the inverse graphics problem of inferring a holistic representation for natural images. Given an input image, our goal is to induce a neuro-symbolic, program-like represen…
Learning Physical Graph Representations from Visual Scenes
Daniel M. Bear, Chaofei Fan, Damian Mrowca +8
Convolutional Neural Networks (CNNs) have proved exceptional at learning representations for visual object categorization. However, CNNs do not explicitly encode objects, parts, an…
DreamCoder: Growing generalizable, interpretable knowledge with wake-sleep Bayesian program learning
Kevin Ellis, Catherine Wong, Maxwell Nye +6
Expert problem-solving is driven by powerful languages for thinking about problems and their solutions. Acquiring expertise means learning these languages -- systems of concepts, a…
Learning with AMIGo: Adversarially Motivated Intrinsic Goals
Andres Campero, Roberta Raileanu, Heinrich Küttler +3
A key challenge for reinforcement learning (RL) consists of learning in environments with sparse extrinsic rewards. In contrast to current RL methods, humans are able to learn new…