414 citations · 1.1k across the 47 of their papers we have counts for
58 papers
Multimodal Unlearning Across Vision, Language, Video, and Audio: Survey of Methods, Datasets, and Benchmarks
Nobin Sarwar, Shubhashis Roy Dipta, Zheyuan Liu +1
With the growing adoption of VLMs, DMs, LLMs, and AFMs, these multimodal foundation models can inadvertently encode sensitive, copyrighted, biased, or unsafe cross-modal associatio…
Benchmarking Algorithmic Bias in Face Recognition: An Experimental Approach Using Synthetic Faces and Human Evaluation
Hao Liang, Pietro Perona, Guha Balakrishnan
We propose an experimental method for measuring bias in face recognition systems. Existing methods to measure bias depend on benchmark datasets that are collected in the wild and a…
Confidence Intervals for Error Rates in 1:1 Matching Tasks: Critical Statistical Analysis and Recommendations
Riccardo Fogliato, Pratik Patil, Pietro Perona
Matching algorithms are commonly used to predict matches between items in a collection. For example, in 1:1 face verification, a matching algorithm predicts whether two face images…
Understanding Label Bias in Single Positive Multi-Label Learning
Julio Arroyo, Pietro Perona, Elijah Cole
Annotating data for multi-label classification is prohibitively expensive because every category of interest must be confirmed to be present or absent. Recent work on single positi…
BKinD-3D: Self-Supervised 3D Keypoint Discovery from Multi-View Videos
Jennifer J. Sun, Lili Karashchuk, Amil Dravid +9
Quantifying motion in 3D is important for studying the behavior of humans and other animals, but manual pose annotations are expensive and time-consuming to obtain. Self-supervised…
Visual Knowledge Tracing
Neehar Kondapaneni, Pietro Perona, Oisin Mac Aodha
Each year, thousands of people learn new visual categorization tasks -- radiologists learn to recognize tumors, birdwatchers learn to distinguish similar species, and crowd workers…
On Label Granularity and Object Localization
Elijah Cole, Kimberly Wilber, Grant Van Horn +6
Weakly supervised object localization (WSOL) aims to learn representations that encode object location using only image-level category labels. However, many objects can be labeled…
The Caltech Fish Counting Dataset: A Benchmark for Multiple-Object Tracking and Counting
Justin Kay, Peter Kulits, Suzanne Stathatos +5
We present the Caltech Fish Counting Dataset (CFC), a large-scale dataset for detecting, tracking, and counting fish in sonar videos. We identify sonar videos as a rich source of d…
Near Perfect GAN Inversion
Qianli Feng, Viraj Shah, Raghudeep Gadde +2
To edit a real photo using Generative Adversarial Networks (GANs), we need a GAN inversion algorithm to identify the latent vector that perfectly reproduces it. Unfortunately, wher…
Towards Weakly-Supervised Text Spotting using a Multi-Task Transformer
Yair Kittenplon, Inbal Lavi, Sharon Fogel +3
Text spotting end-to-end methods have recently gained attention in the literature due to the benefits of jointly optimizing the text detection and recognition components. Existing…
Rayleigh EigenDirections (REDs): GAN latent space traversals for multidimensional features
Guha Balakrishnan, Raghudeep Gadde, Aleix Martinez +1
We present a method for finding paths in a deep generative model's latent space that can maximally vary one set of image features while holding others constant. Crucially, unlike p…
Self-Supervised Keypoint Discovery in Behavioral Videos
Jennifer J. Sun, Serim Ryou, Roni Goldshmid +6
We propose a method for learning the posture and structure of agents from unlabelled behavioral videos. Starting from the observation that behaving agents are generally the main so…
Weakly Supervised Keypoint Discovery
Serim Ryou, Pietro Perona
In this paper, we propose a method for keypoint discovery from a 2D image using image-level supervision. Recent works on unsupervised keypoint discovery reliably discover keypoints…
Multi-Label Learning from Single Positive Labels
Elijah Cole, Oisin Mac Aodha, Titouan Lorieul +3
Predicting all applicable labels for a given image is known as multi-label classification. Compared to the standard multi-class case (where each image has only one label), it is co…
The Multi-Agent Behavior Dataset: Mouse Dyadic Social Interactions
Jennifer J. Sun, Tomomi Karigo, Dipam Chakraborty +8
Multi-agent behavior modeling aims to understand the interactions that occur between agents. We present a multi-agent dataset from behavioral neuroscience, the Caltech Mouse Social…
Matched sample selection with GANs for mitigating attribute confounding
Chandan Singh, Guha Balakrishnan, Pietro Perona
Measuring biases of vision systems with respect to protected attributes like gender and age is critical as these systems gain widespread use in society. However, significant correl…
A linearized framework and a new benchmark for model selection for fine-tuning
Aditya Deshpande, Alessandro Achille, Avinash Ravichandran +6
Fine-tuning from a collection of models pre-trained on different domains (a "model zoo") is emerging as a technique to improve test accuracy in the low-data regime. However, model…
Sequence-to-Sequence Contrastive Learning for Text Recognition
Aviad Aberdam, Ron Litman, Shahar Tsiper +5
We propose a framework for sequence-to-sequence contrastive learning (SeqCLR) of visual representations, which we apply to text recognition. To account for the sequence-to-sequence…
A Number Sense as an Emergent Property of the Manipulating Brain
Neehar Kondapaneni, Pietro Perona
The ability to understand and manipulate numbers and quantities emerges during childhood, but the mechanism through which humans acquire and develop this ability is still poorly un…
Task Programming: Learning Data Efficient Behavior Representations
Jennifer J. Sun, Ann Kennedy, Eric Zhan +3
Specialized domain knowledge is often necessary to accurately annotate training sets for in-depth analysis, but can be burdensome and time-consuming to acquire from domain experts.…
Towards causal benchmarking of bias in face analysis algorithms
Guha Balakrishnan, Yuanjun Xiong, Wei Xia +1
Measuring algorithmic bias is crucial both to assess algorithmic fairness, and to guide the improvement of algorithms. Current methods to measure algorithmic bias in computer visio…
Rethinking Zero-shot Video Classification: End-to-end Training for Realistic Applications
Biagio Brattoli, Joseph Tighe, Fedor Zhdanov +2
Trained on large datasets, deep learning (DL) can accurately classify videos into hundreds of diverse classes. However, video data is expensive to annotate. Zero-shot learning (ZSL…
Geocoding of trees from street addresses and street-level images
Daniel Laumer, Nico Lang, Natalie van Doorn +3
We introduce an approach for updating older tree inventories with geographic coordinates using street-level panorama images and a global optimization framework for tree instance ma…
HMM-guided frame querying for bandwidth-constrained video search
Bhairav Chidambaram, Mason McGill, Pietro Perona
We design an agent to search for frames of interest in video stored on a remote server, under bandwidth constraints. Using a convolutional neural network to score individual frames…