2.3k citations · 2.3k across the 18 of their papers we have counts for
21 papers
TRACER: A Multimodal Harness for Patient Care in Hospitals
Zane Durante, Paul Tang, Keshigeyan Chandrasegaran +11
Continuous video and clinical records provide complementary information for understanding how patient behavior evolves over a hospital stay. A central challenge is how to represent…
Align-RAG: Alignment Is All You Need for TSFM In-Context Learning
Mohammad Asadi, Soheil Hor, Bardiya Akhbari +6
Retrieval-augmented forecasting promises to adapt frozen Time Series Foundation Models (TSFMs) to new domains without fine-tuning, but recent methods typically rely on learned fusi…
Ordered Action Tokens for Visuomotor Policy Learning
Chaoqi Liu, Yue Zhao, Haonan Chen +4
Action tokenization maps continuous robot action chunks to discrete tokens and has become an important interface for modern visuomotor policies. Existing approaches either rely on…
A vision foundation model for single-cell biology via spatial gene cartography
Ridvan Yesiloglu, Sakib Mostafa, James Zou +5
Most single-cell foundation models are adapted from language models, representing each cell as a sequence of gene tokens. This discards the relationships among genes and often the…
HumanScore: Benchmarking Human Motions in Generated Videos
Yusu Fang, Tiange Xiang, Tian Tan +4
Recent advances in model architectures, compute, and data scale have driven rapid progress in video generation, producing increasingly realistic content. Yet, no prior method syste…
Neuro-Symbolic Decoding of Neural Activity
Yanchen Wang, Joy Hsu, Ehsan Adeli +1
We propose NEURONA, a neuro-symbolic framework for fMRI decoding and concept grounding in neural activity. Leveraging image- and video-based fMRI question-answering datasets, NEURO…
Latent Forcing: Reordering the Diffusion Trajectory for Pixel-Space Image Generation
Alan Baade, Eric Ryan Chan, Kyle Sargent +4
Latent diffusion models excel at generating high-quality images but lose the benefits of end-to-end modeling. They discard information during image encoding, require a separately t…
VideoWeave: A Data-Centric Approach for Efficient Video Understanding
Zane Durante, Silky Singh, Arpandeep Khatua +6
Training video-language models is often prohibitively expensive due to the high cost of processing long frame sequences and the limited availability of annotated long videos. We pr…
QuantiPhy: A Quantitative Benchmark Evaluating Physical Reasoning Abilities of Vision-Language Models
Li Puyin, Tiange Xiang, Ella Mao +5
Understanding the physical world is essential for generalist AI agents. However, it remains unclear whether state-of-the-art vision perception models (e.g., large VLMs) can reason…
Repurposing 2D Diffusion Models for 3D Shape Completion
Yao He, Youngjoong Kwon, Tiange Xiang +2
We present a framework that adapts 2D diffusion models for 3D shape completion from incomplete point clouds. While text-to-image diffusion models have achieved remarkable success w…
A Tool Bottleneck Framework for Clinically-Informed and Interpretable Medical Image Understanding
Christina Liu, Alan Q. Wang, Joy Hsu +2
Recent tool-use frameworks powered by vision-language models (VLMs) improve image understanding by grounding model predictions with specialized tools. Broadly, these frameworks lev…
ViBES: A Conversational Agent with Behaviorally-Intelligent 3D Virtual Body
Juze Zhang, Changan Chen, Xin Chen +5
Human communication is inherently multimodal and social: words, prosody, and body language jointly carry intent. Yet most prior systems model human behavior as a translation task c…
T*: Re-thinking Temporal Search for Long-Form Video Understanding
Jinhui Ye, Zihan Wang, Haosen Sun +9
Efficiently understanding long-form videos remains a significant challenge in computer vision. In this work, we revisit temporal search paradigms for long-form video understanding…
Towards Fine-Grained Video Question Answering
Wei Dai, Alan Luo, Zane Durante +5
In the rapidly evolving domain of video understanding, Video Question Answering (VideoQA) remains a focal point. However, existing datasets exhibit gaps in temporal and spatial gra…
Repurposing 2D Diffusion Models with Gaussian Atlas for 3D Generation
Tiange Xiang, Kai Li, Chengjiang Long +5
Recent advances in text-to-image diffusion models have been driven by the increasing availability of paired 2D data. However, the development of 3D diffusion models has been hinder…
The Language of Motion: Unifying Verbal and Non-verbal Language of 3D Human Motion
Changan Chen, Juze Zhang, Shrinidhi K. Lakshmikanth +5
Human communication is inherently multimodal, involving a combination of verbal and non-verbal cues such as speech, facial expressions, and body gestures. Modeling these behaviors…
OccFusion: Rendering Occluded Humans with Generative Diffusion Priors
Adam Sun, Tiange Xiang, Scott Delp +2
Most existing human rendering methods require every part of the human to be fully visible throughout the input video. However, this assumption does not hold in real-life settings w…
An Interactive Agent Foundation Model
Zane Durante, Bidipta Sarkar, Ran Gong +17
The development of artificial intelligence systems is transitioning from creating static, task-specific models to dynamic, agent-based systems capable of performing well in a wide…
Wild2Avatar: Rendering Humans Behind Occlusions
Tiange Xiang, Adam Sun, Scott Delp +3
Rendering the visual appearance of moving humans from occluded monocular videos is a challenging task. Most existing research renders 3D humans under ideal conditions, requiring a…
Rendering Humans from Object-Occluded Monocular Videos
Tiange Xiang, Adam Sun, Jiajun Wu +2
3D understanding and rendering of moving humans from monocular videos is a challenging task. Despite recent progress, the task remains difficult in real-world scenarios, where obst…
On the Opportunities and Risks of Foundation Models
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli +111
AI is undergoing a paradigm shift with the rise of models (e.g., BERT, DALL-E, GPT-3) that are trained on broad data at scale and are adaptable to a wide range of downstream tasks.…