8 citations · 19 across the 21 of their papers we have counts for
25 papers
Eye, Robot: Learning Expressive Gaze Dynamics via a VLM-Guided Transformer-VAE
Yuhang Hu, Junlin Lyu, Yuan Fang +6
Gaze coordinates attention in human interaction through coupled eye, head, and eyelid motion. A humanoid robot must choose where to look while generating a suitable movement toward…
The Smart Buildings Challenge: Learning Foundation Models for Real-World Optimization at Scale
Judah Goldfeder, Ayush Rai, Philippe M. Wyder +11
Buildings cause nearly 40\% of carbon emissions in the US. Of total building emissions, HVAC systems are responsible for approximately 40-60\% and account for 15\% of global energy…
Physics-based Sphere Packing for Lagrangian Mesh Morphing
Jiong Lin, Hod Lipson
This paper studies tetrahedral meshes as the body representation for differentiable simulation and computational design. Fixed-connectivity meshes degrade under large morphs, while…
Mirage Probes: How Vision Models Fake Visual Understanding
Daniel Ben-Levi, Judah Goldfeder, Weiliang Zhao +5
Vision-language models (VLMs) can answer image-based questions confidently, and often correctly, even when no image is provided. This mirage behavior inflates benchmark scores with…
Evidence of an Emergent "Self" in Continual Robot Learning
Adidev Jhunjhunwala, Judah Goldfeder, Hod Lipson
A key challenge to understanding self-awareness has been a principled way of quantifying whether an intelligent system has a concept of a "self", and if so how to differentiate the…
Beyond Cropping and Rotation: Automated Evolution of Powerful Task-Specific Augmentations with Generative Models
Judah Goldfeder, Shreyes Kaliyur, Vaibhav Sourirajan +5
Data augmentation has long been a cornerstone for reducing overfitting in vision models, with methods like AutoAugment automating the design of task-specific augmentations. Recent…
ArticFlow: Generative Simulation of Articulated Mechanisms
Jiong Lin, Jinchen Ruan, Hod Lipson
Recent advances in generative models have produced strong results for static 3D shapes, whereas articulated 3D generation remains challenging due to action-dependent deformations a…
Accelerating scientific discovery with the common task framework
J. Nathan Kutz, Peter Battaglia, Michael Brenner +12
Machine learning (ML) and artificial intelligence (AI) algorithms are transforming and empowering the characterization and control of dynamic systems in the engineering, physical,…
Generating Auxiliary Tasks with Reinforcement Learning
Judah Goldfeder, Matthew So, Hod Lipson
Auxiliary Learning (AL) is a form of multi-task learning in which a model trains on auxiliary tasks to boost performance on a primary objective. While AL has improved generalizatio…
Bi-Encoder Contrastive Learning for Fingerprint and Iris Biometrics
Matthew So, Judah Goldfeder, Mark Lis +1
There has been a historic assumption that the biometrics of an individual are statistically uncorrelated. We test this assumption by training Bi-Encoder networks on three verificat…
LyTimeT: Towards Robust and Interpretable State-Variable Discovery
Kuai Yu, Crystal Su, Xiang Liu +3
Extracting the true dynamical variables of a system from high-dimensional video is challenging due to distracting visual factors such as background motion, occlusions, and texture…
Creative synthesis of kinematic mechanisms
Jiong Lin, Jialong Ning, Judah Goldfeder +1
In this paper, we formulate the problem of kinematic synthesis for planar linkages as a cross-domain image generation task. We develop a planar linkages dataset using RGB image rep…
AutoURDF: Unsupervised Robot Modeling from Point Cloud Frames Using Cluster Registration
Jiong Lin, Lechen Zhang, Kwansoo Lee +3
Robot description models are essential for simulation and control, yet their creation often requires significant manual effort. To streamline this modeling process, we introduce Au…
Robot Metabolism: Towards machines that can grow by consuming other machines
Philippe Martin Wyder, Riyaan Bakhda, Meiqi Zhao +17
Biological lifeforms can heal, grow, adapt, and reproduce -- abilities essential for sustained survival and development. In contrast, robots today are primarily monolithic machines…
The Smart Buildings Control Suite: A Diverse Open Source Benchmark to Evaluate and Scale HVAC Control Policies for Sustainability
Judah Goldfeder, Victoria Dean, Zixin Jiang +4
Commercial buildings account for 17% of U.S. carbon emissions, with roughly half of that from Heating, Ventilation, and Air Conditioning (HVAC). HVAC devices form a complex thermod…
Sequencing the Neurome: Towards Scalable Exact Parameter Reconstruction of Black-Box Neural Networks
Judah Goldfeder, Quinten Roets, Gabe Guo +2
Inferring the exact parameters of a neural network with only query access is an NP-Hard problem, with few practical existing algorithms. Solutions would have major implications for…
Ab Initio Structure Solutions from Nanocrystalline Powder Diffraction Data
Gabe Guo, Tristan Saidi, Maxwell Terban +3
A major challenge in materials science is the determination of the structure of nanometer sized objects. Here we present a novel approach that uses a generative machine learning mo…
Reconfigurable Robot Identification from Motion Data
Yuhang Hu, Yunzhe Wang, Ruibo Liu +2
Integrating Large Language Models (VLMs) and Vision-Language Models (VLMs) with robotic systems enables robots to process and understand complex natural language instructions and v…
Towards End-to-End Structure Solutions from Information-Compromised Diffraction Data via Generative Deep Learning
Gabe Guo, Judah Goldfeder, Ling Lan +5
The revolution in materials in the past century was built on a knowledge of the atomic arrangements and the structure-property relationship. The sine qua non for obtaining quantita…
Balanced and Deterministic Weight-sharing Helps Network Performance
Oscar Chang, Hod Lipson
Weight-sharing plays a significant role in the success of many deep neural networks, by increasing memory efficiency and incorporating useful inductive priors about the problem int…
Accelerating Meta-Learning by Sharing Gradients
Oscar Chang, Hod Lipson
The success of gradient-based meta-learning is primarily attributed to its ability to leverage related tasks to learn task-invariant information. However, the absence of interactio…
Teaching Robots to Build Simulations of Themselves
Yuhang Hu, Jiong Lin, Hod Lipson
The emergence of vision catalysed a pivotal evolutionary advancement, enabling organisms not only to perceive but also to interact intelligently with their environment. This transf…
Knolling bot 2.0: Enhancing Object Organization with Self-supervised Graspability Estimation
Yuhang Hu, Zhizhuo Zhang, Hod Lipson
Building on recent advancements in transformer based approaches for domestic robots performing knolling, the art of organizing scattered items into neat arrangements. This paper in…
Knolling Bot: Teaching Robots the Human Notion of Tidiness
Yuhang Hu, Judah Goldfeder, Zhizhuo Zhang +5
For robots to truly collaborate and assist humans, they must understand not only logic and instructions, but also the subtle emotions, aesthetics, and feelings that define our huma…