4 papers
3D-Generalist: Self-Improving Vision-Language-Action Models for Crafting 3D Worlds
Fan-Yun Sun, Shengguang Wu, Christian Jacobsen +13
Despite large-scale pretraining endowing models with language and vision reasoning capabilities, improving their spatial reasoning capability remains challenging due to the lack of…
Fly, Fail, Fix: Iterative Game Repair with Reinforcement Learning and Large Multimodal Models
Alex Zook, Josef Spjut, Jonathan Tremblay
Game design hinges on understanding how static rules and content translate into dynamic player behavior - something modern generative systems that inspect only a game's code or ass…
GRS: Generating Robotic Simulation Tasks from Real-World Images
Alex Zook, Fan-Yun Sun, Josef Spjut +3
We introduce GRS (Generating Robotic Simulation tasks), a system addressing real-to-sim for robotic simulations. GRS creates digital twin simulations from single RGB-D observations…
FactorSim: Generative Simulation via Factorized Representation
Fan-Yun Sun, S. I. Harini, Angela Yi +6
Generating simulations to train intelligent agents in game-playing and robotics from natural language input, from user input or task documentation, remains an open-ended challenge.…