11 citations · 13 across the 10 of their papers we have counts for
11 papers
Language-Guided Terrain-Adaptive Neural MPC for Autonomous Traversal of Articulated Tracked Robots
Zhenfeng Gan, Yanbo Chen, Lirong Che +4
In urban search and rescue, articulated tracked robots (ATRs) must traverse structured but contact-rich environments such as stairwells and cluttered building interiors. Reliable a…
ActSWM: Action-Sensitive World Models for Long-Horizon Planning in Open-World Games
Zhenfeng Gan, ZiTong Zeng, Jiajun Cheng +3
Latent world models support efficient model-predictive control by optimizing future control sequences in latent space and replanning in a receding-horizon manner. However, existing…
PhotoAgent: A Robotic Photographer with Spatial and Aesthetic Understanding
Lirong Che, Zhenfeng Gan, Yanbo Chen +2
Embodied agents for creative tasks like photography must bridge the semantic gap between high-level language commands and geometric control. We introduce PhotoAgent, an agent that…
Learning Structured Reasoning via Tractable Trajectory Control
Po-Nien Kung, Zhen Yang, Jeffrey Luo +7
Large language models can exhibit emergent reasoning behaviors, often manifested as recurring lexical patterns (e.g., "wait," indicating verification). However, complex reasoning t…
UltraCUA: A Foundation Model for Computer Use Agents with Hybrid Action
Yuhao Yang, Zhen Yang, Zi-Yi Dou +10
Computer-use agents face a fundamental limitation. They rely exclusively on primitive GUI actions (click, type, scroll), creating brittle execution chains prone to cascading failur…
Ferret-UI Lite: Lessons from Building Small On-Device GUI Agents
Zhen Yang, Zi-Yi Dou, Di Feng +13
Developing autonomous agents that effectively interact with Graphic User Interfaces (GUIs) remains a challenging open problem, especially for small on-device models. In this paper,…
Scaling Synthetic Task Generation for Agents via Exploration
Ram Ramrakhya, Andrew Szot, Omar Attia +6
Post-Training Multimodal Large Language Models (MLLMs) to build interactive agents holds promise across domains such as computer-use, web navigation, and robotics. A key challenge…
From Multimodal LLMs to Generalist Embodied Agents: Methods and Lessons
Andrew Szot, Bogdan Mazoure, Omar Attia +6
We examine the capability of Multimodal Large Language Models (MLLMs) to tackle diverse domains that extend beyond the traditional language and vision tasks these models are typica…
Multimodal Autoregressive Pre-training of Large Vision Encoders
Enrico Fini, Mustafa Shukor, Xiujun Li +13
We introduce a novel method for pre-training of large-scale vision encoders. Building on recent advancements in autoregressive pre-training of vision models, we extend this framewo…
MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training
Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier +29
In this work, we discuss building performant Multimodal Large Language Models (MLLMs). In particular, we study the importance of various architecture components and data choices. T…
Pre-trained Language Models Do Not Help Auto-regressive Text-to-Image Generation
Yuhui Zhang, Brandon McKinzie, Zhe Gan +2
Recent advances in image tokenizers, such as VQ-VAE, have enabled text-to-image generation using auto-regressive methods, similar to language modeling. However, these methods have…