6 papers
ToolGrad: Efficient Tool-use Dataset Generation with Textual "Gradients"
Zhongyi Zhou, Kohei Uehara, Haoyu Zhang +5
Prior work synthesizes tool-use LLM datasets by first generating a user query, followed by complex tool-use annotations like depth-first search (DFS). This leads to inevitable anno…
Vibe Coding XR: Accelerating AI + XR Prototyping with XR Blocks and Gemini
Ruofei Du, Benjamin Hersh, David Li +11
While large language models (LLMs) have accelerated 2D software development through intent-driven "vibe coding", prototyping intelligent Extended Reality (XR) experiences remains a…
Exploring the Role of User Comments Throughout the Stages of Video-Based Task-Learning
Nayoung Kim, Yotam Sechayk, Zhongyi Zhou +1
Learning tasks through videos is a dynamic way to acquire skills by witnessing entire processes. However, compared to in-person demonstrations, videos may omit tacit knowledge, inc…
XR Blocks: Accelerating Human-centered AI + XR Innovation
David Li, Nels Numan, Xun Qian +17
We are on the cusp where Artificial Intelligence (AI) and Extended Reality (XR) are converging to unlock new paradigms of interactive computing. However, a significant gap exists b…
Vision-Based Multimodal Interfaces: A Survey and Taxonomy for Enhanced Context-Aware System Design
Yongquan 'Owen' Hu, Jingyu Tang, Xinya Gong +6
The recent surge in artificial intelligence, particularly in multimodal processing technology, has advanced human-computer interaction, by altering how intelligent systems perceive…
InstructPipe: Generating Visual Blocks Pipelines with Human Instructions and LLMs
Zhongyi Zhou, Jing Jin, Vrushank Phadnis +16
Visual programming has the potential of providing novice programmers with a low-code experience to build customized processing pipelines. Existing systems typically require users t…