62 citations · 176 across the 28 of their papers we have counts for
33 papers
Providing Rapid Design Feedback for 3D Obstacle Course Games Using Constrained Solvability Queries
Zander Majercik, Sharon Zhang, William Wang +6
We present a system that aids the design of 3D obstacle course games by providing designers with rapid feedback on how obstacles can be solved. Our core contribution is a system fo…
Learning Realistic Athletic Sprinting Without Demonstrations
William Wang, Nicholas Bianco, Guy Tevet +4
We present a muscle-driven simulation system for generating biomechanically accurate motion for high-speed athletic locomotion tasks that does not require motion demonstrations. Ou…
Inference-Time Distillation: Cost-Efficient Agents Without Fine-Tuning or Manual Prompt Engineering
Vishnu Sarukkai, Asanshay Gupta, James Hong +2
Deploying LLM agents at scale typically requires choosing between quality and cost. Existing cost-reduction approaches fail to preserve agility: the ability to iterate rapidly with…
Bonsai: Compiling Queries to Pruned Tree Traversals
Alexander J Root, Christophe Gyurgyik, Purvi Goel +4
Trees can accelerate queries that search or aggregate values over large collections. They achieve this by storing metadata that enables quick pruning (or inclusion) of subtrees whe…
Learning to Ball: Composing Policies for Long-Horizon Basketball Moves
Pei Xu, Zhen Wu, Ruocheng Wang +5
Learning a control policy for a multi-phase, long-horizon task, such as basketball maneuvers, remains challenging for reinforcement learning approaches due to the need for seamless…
Generating Detailed Character Motion from Blocking Poses
Purvi Goel, Guy Tevet, C. K. Liu +1
We focus on the problem of using generative diffusion models for the task of motion detailing: converting a rough version of a character animation, represented by a sparse set of c…
Self-Generated In-Context Examples Improve LLM Agents for Sequential Decision-Making Tasks
Vishnu Sarukkai, Zhiqiang Xie, Kayvon Fatahalian
Improving Large Language Model (LLM) agents for sequential decision-making tasks typically requires extensive task-specific knowledge engineering--custom prompts, curated examples,…
Generative Motion Infilling From Imprecisely Timed Keyframes
Purvi Goel, Haotian Zhang, C. Karen Liu +1
Keyframes are a standard representation for kinematic motion specification. Recent learned motion-inbetweening methods use keyframes as a way to control generative motion models, a…
Cube: A Roblox View of 3D Intelligence
Foundation AI Team, Kiran Bhat, Nishchaie Khanna +44
Foundation models trained on vast amounts of data have demonstrated remarkable reasoning and generation capabilities in the domains of text, images, audio and video. Our goal at Ro…
AI Metropolis: Scaling Large Language Model-based Multi-Agent Simulation with Out-of-order Execution
Zhiqiang Xie, Hao Kang, Ying Sheng +3
With more advanced natural language understanding and reasoning capabilities, large language model (LLM)-powered agents are increasingly developed in simulated environments to perf…
Automated Rewards via LLM-Generated Progress Functions
Vishnu Sarukkai, Brennan Shacklett, Zander Majercik +3
Large Language Models (LLMs) have the potential to automate reward engineering by leveraging their broad domain knowledge across various tasks. However, they often need many iterat…
Learning to Move Like Professional Counter-Strike Players
David Durst, Feng Xie, Vishnu Sarukkai +9
In multiplayer, first-person shooter games like Counter-Strike: Global Offensive (CS:GO), coordinated movement is a critical component of high-level strategic play. However, the co…
Block and Detail: Scaffolding Sketch-to-Image Generation
Vishnu Sarukkai, Lu Yuan, Mia Tang +2
We introduce a novel sketch-to-image tool that aligns with the iterative refinement process of artists. Our tool lets users sketch blocking strokes to coarsely represent the placem…
Learning Subject-Aware Cropping by Outpainting Professional Photos
James Hong, Lu Yuan, Michaël Gharbi +2
How to frame (or crop) a photo often depends on the image subject and its context; e.g., a human portrait. Recent works have defined the subject-aware image cropping task as a nuan…
Iterative Motion Editing with Natural Language
Purvi Goel, Kuan-Chieh Wang, C. Karen Liu +1
Text-to-motion diffusion models can generate realistic animations from text prompts, but do not support fine-grained motion editing controls. In this paper, we present a method for…
Collage Diffusion
Vishnu Sarukkai, Linden Li, Arden Ma +2
We seek to give users precise control over diffusion-based image generation by modeling complex scenes as sequences of layers, which define the desired spatial arrangement and visu…
Spotting Temporally Precise, Fine-Grained Events in Video
James Hong, Haotian Zhang, Michaël Gharbi +2
We introduce the task of spotting temporally precise, fine-grained events in video (detecting the precise moment in time events occur). Precise spotting requires models to reason g…
Perfectly Balanced: Improving Transfer and Robustness of Supervised Contrastive Learning
Mayee F. Chen, Daniel Y. Fu, Avanika Narayan +4
An ideal learned representation should display transferability and robustness. Supervised contrastive learning (SupCon) is a promising method for training accurate models, but prod…
Shoring Up the Foundations: Fusing Model Embeddings and Weak Supervision
Mayee F. Chen, Daniel Y. Fu, Dyah Adila +4
Foundation models offer an exciting new paradigm for constructing models with out-of-the-box embeddings and a few labeled examples. However, it is not clear how to best apply found…
Low-Shot Validation: Active Importance Sampling for Estimating Classifier Performance on Rare Categories
Fait Poms, Vishnu Sarukkai, Ravi Teja Mullapudi +4
For machine learning models trained with limited labeled training data, validation stands to become the main bottleneck to reducing overall annotation costs. We propose a statistic…
Video Pose Distillation for Few-Shot, Fine-Grained Sports Action Recognition
James Hong, Matthew Fisher, Michaël Gharbi +1
Human pose is a useful feature for fine-grained sports action understanding. However, pose estimators are often unreliable when run on sports video due to domain shift and factors…
Mandoline: Model Evaluation under Distribution Shift
Mayee Chen, Karan Goel, Nimit S. Sohoni +3
Machine learning models are often deployed in different settings than they were trained and validated on, posing a challenge to practitioners who wish to predict how well the deplo…
Large Batch Simulation for Deep Reinforcement Learning
Brennan Shacklett, Erik Wijmans, Aleksei Petrenko +4
We accelerate deep reinforcement learning-based training in visually complex 3D environments by two orders of magnitude over prior work, realizing end-to-end training speeds of ove…
Iterative Text-based Editing of Talking-heads Using Neural Retargeting
Xinwei Yao, Ohad Fried, Kayvon Fatahalian +1
We present a text-based tool for editing talking-head video that enables an iterative editing workflow. On each iteration users can edit the wording of the speech, further refine m…