Publications (8)
Measuring Self-Supervised Representation Quality for Downstream Classification using Discriminative Features
Neha Kalibhat, Kanika Narang, Hamed Firooz +2
Self-supervised learning (SSL) has shown impressive results in downstream classification tasks. However, there is limited work in understanding their failure modes and interpreting…
Understanding the Effect of using Semantically Meaningful Tokens for Visual Representation Learning
Neha Kalibhat, Priyatham Kattakinda, Sumit Nawathe +5
Vision transformers have established a precedent of patchifying images into uniformly-sized chunks before processing. We hypothesize that this design choice may limit models in lea…
Visual prompt engineering for video models
Robert Geirhos, Yuxuan Li, Thaddäus Wiedemer +7
In the age of foundation models, a model is only as good as its prompt. For this reason, prompt engineering has become an essential technique for improving language model performan…
Adapting Self-Supervised Representations to Multi-Domain Setups
Neha Kalibhat, Sam Sharpe, Jeremy Goodsitt +2
Current state-of-the-art self-supervised approaches, are effective when trained on individual domains but show limited generalization on unseen domains. We observe that these model…
Augmentations vs Algorithms: What Works in Self-Supervised Learning
Warren Morningstar, Alex Bijamov, Chris Duvarney +8
We study the relative effects of data augmentations, pretraining algorithms, and model architectures in Self-Supervised Learning (SSL). While the recent literature in this space le…
Identifying Interpretable Subspaces in Image Representations
Neha Kalibhat, Shweta Bhardwaj, Bayan Bruss +3
We propose Automatic Feature Explanation using Contrasting Concepts (FALCON), an interpretability framework to explain features of image representations. For a target feature, FALC…
Interpreting and Controlling Model Behavior via Constitutions for Atomic Concept Edits
Neha Kalibhat, Zi Wang, Prasoon Bajpai +4
We introduce a black-box interpretability framework that learns a verifiable constitution: a natural language summary of how changes to a prompt affect a model's specific behavior,…
Disentangling the Effects of Data Augmentation and Format Transform in Self-Supervised Learning of Image Representations
Neha Kalibhat, Warren Morningstar, Alex Bijamov +3
Self-Supervised Learning (SSL) enables training performant models using limited labeled data. One of the pillars underlying vision SSL is the use of data augmentations/perturbation…