Publications (14)
PiCIE: Unsupervised Semantic Segmentation using Invariance and Equivariance in Clustering
Jang Hyun Cho, Utkarsh Mall, Kavita Bala +1
We present a new framework for semantic segmentation without annotations via clustering. Off-the-shelf clustering methods are limited to curated, single-label, and object-centric i…
Field-Guide-Inspired Zero-Shot Learning
Utkarsh Mall, Bharath Hariharan, Kavita Bala
Modern recognition systems require large amounts of supervision to achieve accuracy. Adapting to new domains requires significant data from experts, which is onerous and can become…
Flat-Pack Bench: Evaluating Spatio-Temporal Understanding in Large Vision-Language Models through Furniture Assembly
Aditya Chetan, Eric Cai, Peeyush Kushwaha +5
The emergence of Large Vision-Language Models (LVLMs) has significantly advanced video understanding capabilities. However, existing benchmarks focus predominantly on coarse-graine…
Discovering Underground Maps from Fashion
Utkarsh Mall, Kavita Bala, Tamara Berg +1
The fashion sense -- meaning the clothing styles people wear -- in a geographical region can reveal information about that region. For example, it can reflect the kind of activitie…
Towards LLM Agents for Earth Observation
Chia Hsiang Kao, Wenting Zhao, Shreelekha Revankar +8
Earth Observation (EO) provides critical planetary data for environmental monitoring, disaster management, climate science, and other scientific domains. Here we ask: Are AI system…
How Video Meetings Change Your Expression
Sumit Sarin, Utkarsh Mall, Purva Tendulkar +1
Do our facial expressions change when we speak over video calls? Given two unpaired sets of videos of people, we seek to automatically find spatio-temporal patterns that are distin…
AllClear: A Comprehensive Dataset and Benchmark for Cloud Removal in Satellite Imagery
Hangyu Zhou, Chia-Hsiang Kao, Cheng Perng Phoo +3
Clouds in satellite imagery pose a significant challenge for downstream applications. A major challenge in current cloud removal research is the absence of a comprehensive benchmar…
Remote Sensing Vision-Language Foundation Models without Annotations via Ground Remote Alignment
Utkarsh Mall, Cheng Perng Phoo, Meilin Kelsey Liu +3
We introduce a method to train vision-language models for remote-sensing images without using any textual annotations. Our key insight is to use co-located internet imagery taken o…
MONITRS: Multimodal Observations of Natural Incidents Through Remote Sensing
Shreelekha Revankar, Utkarsh Mall, Cheng Perng Phoo +2
Natural disasters cause devastating damage to communities and infrastructure every year. Effective disaster response is hampered by the difficulty of accessing affected areas durin…
GeoStyle: Discovering Fashion Trends and Events
Utkarsh Mall, Kevin Matzen, Bharath Hariharan +2
Understanding fashion styles and trends is of great potential interest to retailers and consumers alike. The photos people upload to social media are a historical and public data s…
Scale-Aware Recognition in Satellite Images under Resource Constraints
Shreelekha Revankar, Cheng Perng Phoo, Utkarsh Mall +2
Recognition of features in satellite imagery (forests, swimming pools, etc.) depends strongly on the spatial scale of the concept and therefore the resolution of the images. This p…
Evolving Interpretable Visual Classifiers with Large Language Models
Mia Chiquier, Utkarsh Mall, Carl Vondrick
Multimodal pre-trained models, such as CLIP, are popular for zero-shot classification due to their open-vocabulary flexibility and high performance. However, vision-language models…
A Deep Recurrent Framework for Cleaning Motion Capture Data
Utkarsh Mall, G. Roshan Lal, Siddhartha Chaudhuri +1
We present a deep, bidirectional, recurrent framework for cleaning noisy and incomplete motion capture data. It exploits temporal coherence and joint correlations to infer adaptive…
DiSciPLE: Learning Interpretable Programs for Scientific Visual Discovery
Utkarsh Mall, Cheng Perng Phoo, Mia Chiquier +3
Visual data is used in numerous different scientific workflows ranging from remote sensing to ecology. As the amount of observation data increases, the challenge is not just to mak…