8 citations · 16 across the 22 of their papers we have counts for
24 papers
GeONet: Geometry-Encoded Operator Network for Predicting Flow in Heterogeneous Porous Media
Md Lal Mamud, Maruti K. Mudunuru, Yucheng Fu +13
Rapid prediction of pressure-driven flow in microfluidic porous media is important for experimental design, tracer-test interpretation, and digital-twin development for subsurface…
Discriminative World Models for Web Agents
Kelvin Li, Dhruv Pendharkar, Anish Pahilajani +6
Recent web agents use world models for test-time action selection by sampling candidate actions, predicting the resulting web states, and ranking them with a ranker model or a Proc…
EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data
Ruijie Zheng, Dantong Niu, Yuqi Xie +12
Human behavior is among the most scalable sources of data for learning physical intelligence, yet how to effectively leverage it for dexterous manipulation remains unclear. While p…
DAVE: A VLM Vision Encoder for Document Understanding and Web Agents
Brandon Huang, Hang Hua, Zhuoran Yu +3
While Vision-language models (VLMs) have demonstrated remarkable performance across multi-modal tasks, their choice of vision encoders presents a fundamental weakness: their low-le…
Latent Implicit Visual Reasoning
Kelvin Li, Chuyi Shang, Leonid Karlinsky +3
While Large Multimodal Models (LMMs) have made significant progress, they remain largely text-centric, relying on language as their core reasoning modality. As a result, they are l…
Mechanistic Finetuning of Vision-Language-Action Models via Few-Shot Demonstrations
Chancharik Mitra, Yusen Luo, Raj Saravanan +7
Vision-Language Action (VLAs) models promise to extend the remarkable success of vision-language models (VLMs) to robotics. Yet, unlike VLMs in the vision-language domain, VLAs for…
Learning to Grasp Anything by Playing with Random Toys
Dantong Niu, Yuvan Sharma, Baifeng Shi +11
Robotic manipulation policies often struggle to generalize to novel objects, limiting their real-world utility. In contrast, cognitive science suggests that children develop genera…
Do What? Teaching Vision-Language-Action Models to Reject the Impossible
Wen-Han Hsieh, Elvis Hsieh, Dantong Niu +3
Recently, Vision-Language-Action (VLA) models have demonstrated strong performance on a range of robotic tasks. These models rely on multimodal inputs, with language instructions p…
Activation Reward Models for Few-Shot Model Alignment
Tianning Chai, Chancharik Mitra, Brandon Huang +8
Aligning Large Language Models (LLMs) and Large Multimodal Models (LMMs) to human preferences is a central challenge in improving the quality of the models' generative outputs for…
LeVERB: Humanoid Whole-Body Control with Latent Vision-Language Instruction
Haoru Xue, Xiaoyu Huang, Dantong Niu +8
Vision-language-action (VLA) models have demonstrated strong semantic understanding and zero-shot generalization, yet most existing systems assume an accurate low-level controller…
Visualizing Thought: Conceptual Diagrams Enable Robust Planning in LMMs
Nasim Borazjanizadeh, Roei Herzig, Eduard Oks +3
Human reasoning relies on constructing and manipulating mental models -- simplified internal representations of situations used to understand and solve problems. Conceptual diagram…
Pre-training Auto-regressive Robotic Models with 4D Representations
Dantong Niu, Yuvan Sharma, Haoru Xue +5
Foundation models pre-trained on massive unlabeled datasets have revolutionized natural language and computer vision, exhibiting remarkable generalization capabilities, thus highli…
Enhancing Few-Shot Vision-Language Classification with Large Multimodal Model Features
Chancharik Mitra, Brandon Huang, Tianning Chai +7
Generative Large Multimodal Models (LMMs) like LLaVA and Qwen-VL excel at a wide variety of vision-language (VL) tasks. Despite strong performance, LMMs' generative outputs are not…
In-Context Learning Enables Robot Action Prediction in LLMs
Yida Yin, Zekai Wang, Yuvan Sharma +3
Recently, Large Language Models (LLMs) have achieved remarkable success using in-context learning (ICL) in the language domain. However, leveraging the ICL capabilities within LLMs…
LLARVA: Vision-Action Instruction Tuning Enhances Robot Learning
Dantong Niu, Yuvan Sharma, Giscard Biamby +5
In recent years, instruction-tuned Large Multimodal Models (LMMs) have been successful at several tasks, including image captioning and visual question answering; yet leveraging th…
Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning
Brandon Huang, Chancharik Mitra, Assaf Arbelle +3
The recent success of interleaved Large Multimodal Models (LMMs) in few-shot learning suggests that in-context learning (ICL) with many examples can be promising for learning new t…
Navigating the Labyrinth: Evaluating LLMs' Ability to Reason About Search Problems
Nasim Borazjanizadeh, Roei Herzig, Trevor Darrell +2
Large Language Models (LLMs) have recently achieved impressive performance in math and reasoning benchmarks. However, they often struggle with logic problems and puzzles that are r…
TraveLER: A Modular Multi-LMM Agent Framework for Video Question-Answering
Chuyi Shang, Amos You, Sanjay Subramanian +2
Recently, image-based Large Multimodal Models (LMMs) have made significant progress in video question-answering (VideoQA) using a frame-wise approach by leveraging large-scale pret…
From Audio to Photoreal Embodiment: Synthesizing Humans in Conversations
Evonne Ng, Javier Romero, Timur Bagautdinov +4
We present a framework for generating full-bodied photorealistic avatars that gesture according to the conversational dynamics of a dyadic interaction. Given speech audio, we outpu…
Unsupervised Universal Image Segmentation
Dantong Niu, Xudong Wang, Xinyang Han +3
Several unsupervised image segmentation approaches have been proposed which eliminate the need for dense manually-annotated segmentation masks; current models separately handle eit…
Object-based (yet Class-agnostic) Video Domain Adaptation
Dantong Niu, Amir Bar, Roei Herzig +2
Existing video-based action recognition systems typically require dense annotation and struggle in environments when there is significant distribution shift relative to the trainin…
Compositional Chain-of-Thought Prompting for Large Multimodal Models
Chancharik Mitra, Brandon Huang, Trevor Darrell +1
The combination of strong visual backbones and Large Language Model (LLM) reasoning has led to Large Multimodal Models (LMMs) becoming the current standard for a wide range of visi…
Which One? Leveraging Context Between Objects and Multiple Views for Language Grounding
Chancharik Mitra, Abrar Anwar, Rodolfo Corona +3
When connecting objects and their language referents in an embodied 3D environment, it is important to note that: (1) an object can be better characterized by leveraging comparativ…
Bringing Image Scene Structure to Video via Frame-Clip Consistency of Object Tokens
Elad Ben-Avraham, Roei Herzig, Karttikeya Mangalam +5
Recent action recognition models have achieved impressive results by integrating objects, their locations and interactions. However, obtaining dense structured annotations for each…