38 citations · 190 across the 20 of their papers we have counts for
23 papers
Do multimodal models imagine electric sheep?
Santhosh Kumar Ramakrishnan, Carl Vondrick, Raja Giryes +2
Yes. We find that large multimodal models develop mental imagery when solving spatial puzzles, and they do imagine sheep when solving sheep puzzles. We fine-tune a Qwen3.5 VLM to s…
Does Spatial Cognition Emerge in Frontier Models?
Santhosh Kumar Ramakrishnan, Erik Wijmans, Philipp Kraehenbuehl +1
Not yet. We present SPACE, a benchmark that systematically evaluates spatial cognition in frontier models. Our benchmark builds on decades of research in cognitive science. It eval…
Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives
Kristen Grauman, Andrew Westbury, Lorenzo Torresani +98
We present Ego-Exo4D, a diverse, large-scale multimodal multiview video dataset and benchmark challenge. Ego-Exo4D centers around simultaneously-captured egocentric and exocentric…
Video-Mined Task Graphs for Keystep Recognition in Instructional Videos
Kumar Ashutosh, Santhosh Kumar Ramakrishnan, Triantafyllos Afouras +1
Procedural activity understanding requires perceiving human actions in terms of a broader task, where multiple keysteps are performed in sequence across a long video to reach a fin…
SpotEM: Efficient Video Search for Episodic Memory
Santhosh Kumar Ramakrishnan, Ziad Al-Halah, Kristen Grauman
The goal in episodic memory (EM) is to search a long egocentric video to answer a natural language query (e.g., "where did I leave my purse?"). Existing EM methods exhaustively ext…
Single-Stage Visual Query Localization in Egocentric Videos
Hanwen Jiang, Santhosh Kumar Ramakrishnan, Kristen Grauman
Visual Query Localization on long-form egocentric videos requires spatio-temporal search and localization of visually specified objects and is vital to build episodic memory system…
A Domain-Agnostic Approach for Characterization of Lifelong Learning Systems
Megan M. Baker, Alexander New, Mario Aguilar-Simon +44
Despite the advancement of machine learning techniques in recent years, state-of-the-art systems lack robustness to "real world" events, where the input distributions and tasks enc…
NaQ: Leveraging Narrations as Queries to Supervise Episodic Memory
Santhosh Kumar Ramakrishnan, Ziad Al-Halah, Kristen Grauman
Searching long egocentric videos with natural language queries (NLQ) has compelling applications in augmented reality and robotics, where a fluid index into everything that a perso…
Habitat-Matterport 3D Semantics Dataset
Karmesh Yadav, Ram Ramrakhya, Santhosh Kumar Ramakrishnan +9
We present the Habitat-Matterport 3D Semantics (HM3DSEM) dataset. HM3DSEM is the largest dataset of 3D real-world spaces with densely annotated semantics that is currently availabl…
EgoEnv: Human-centric environment representations from egocentric video
Tushar Nagarajan, Santhosh Kumar Ramakrishnan, Ruta Desai +2
First-person video highlights a camera-wearer's activities in the context of their persistent environment. However, current video understanding approaches reason over visual featur…
Zero Experience Required: Plug & Play Modular Transfer Learning for Semantic Visual Navigation
Ziad Al-Halah, Santhosh K. Ramakrishnan, Kristen Grauman
In reinforcement learning for visual navigation, it is common to develop a model for each new task, and train that model from scratch with task-specific interactions in 3D environm…
PONI: Potential Functions for ObjectGoal Navigation with Interaction-free Learning
Santhosh Kumar Ramakrishnan, Devendra Singh Chaplot, Ziad Al-Halah +2
State-of-the-art approaches to ObjectGoal navigation rely on reinforcement learning and typically require significant computational resources and time for learning. We propose Pote…
Ego4D: Around the World in 3,000 Hours of Egocentric Video
Kristen Grauman, Andrew Westbury, Eugene Byrne +82
We introduce Ego4D, a massive-scale egocentric video dataset and benchmark suite. It offers 3,670 hours of daily-life activity video spanning hundreds of scenarios (household, outd…
Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AI
Santhosh K. Ramakrishnan, Aaron Gokaslan, Erik Wijmans +10
We present the Habitat-Matterport 3D (HM3D) dataset. HM3D is a large-scale dataset of 1,000 building-scale 3D reconstructions from a diverse set of real-world locations. Each scene…
Shapes as Product Differentiation: Neural Network Embedding in the Analysis of Markets for Fonts
Sukjin Han, Eric H. Schulman, Kristen Grauman +1
Many differentiated products have key attributes that are unstructured and thus high-dimensional (e.g., design, text). Instead of treating unstructured attributes as unobservables…
Environment Predictive Coding for Embodied Agents
Santhosh K. Ramakrishnan, Tushar Nagarajan, Ziad Al-Halah +1
We introduce environment predictive coding, a self-supervised approach to learn environment-level representations for embodied agents. In contrast to prior work on self-supervised…
Occupancy Anticipation for Efficient Exploration and Navigation
Santhosh K. Ramakrishnan, Ziad Al-Halah, Kristen Grauman
State-of-the-art navigation methods leverage a spatial memory to generalize to new environments, but their occupancy maps are limited to capturing the geometric structures directly…
Learning to Set Waypoints for Audio-Visual Navigation
Changan Chen, Sagnik Majumder, Ziad Al-Halah +3
In audio-visual navigation, an agent intelligently travels through a complex, unmapped 3D environment using both sights and sounds to find a sound source (e.g., a phone ringing in…
An Exploration of Embodied Visual Exploration
Santhosh K. Ramakrishnan, Dinesh Jayaraman, Kristen Grauman
Embodied computer vision considers perception for robots in novel, unstructured environments. Of particular importance is the embodied visual exploration problem: how might a robot…
Emergence of Exploratory Look-Around Behaviors through Active Observation Completion
Santhosh K. Ramakrishnan, Dinesh Jayaraman, Kristen Grauman
Standard computer vision systems assume access to intelligently captured inputs (e.g., photos from a human photographer), yet autonomously capturing good observations is a major ch…
Sidekick Policy Learning for Active Visual Exploration
Santhosh K. Ramakrishnan, Kristen Grauman
We consider an active visual exploration scenario, where an agent must intelligently select its camera motions to efficiently reconstruct the full environment from only a limited s…
CoMaL Tracking: Tracking Points at the Object Boundaries
Santhosh K. Ramakrishnan, Swarna Kamlam Ravindran, Anurag Mittal
Traditional point tracking algorithms such as the KLT use local 2D information aggregation for feature detection and tracking, due to which their performance degrades at the object…
An Empirical Evaluation of Visual Question Answering for Novel Objects
Santhosh K. Ramakrishnan, Ambar Pal, Gaurav Sharma +1
We study the problem of answering questions about images in the harder setting, where the test questions and corresponding images contain novel objects, which were not queried abou…