18 citations · 44 across the 17 of their papers we have counts for
17 papers
How to Merge Your Multimodal Models Over Time?
Sebastian Dziadzio, Vishaal Udandarao, Karsten Roth +4
Model merging combines multiple expert models - finetuned from a base foundation model on diverse tasks and domains - into a single, more capable model. However, most existing mode…
ONEBench to Test Them All: Sample-Level Benchmarking Over Open-Ended Capabilities
Adhiraj Ghosh, Sebastian Dziadzio, Ameya Prabhu +3
Traditional fixed test sets fall short in evaluating open-ended capabilities of foundation models. To address this, we propose ONEBench(OpeN-Ended Benchmarking), a new testing para…
Active Data Curation Effectively Distills Large-Scale Multimodal Models
Vishaal Udandarao, Nikhil Parthasarathy, Muhammad Ferjad Naeem +6
Knowledge distillation (KD) is the de facto standard for compressing large-scale models into smaller ones. Prior works have explored ever more complex KD strategies involving diffe…
Centaur: a foundation model of human cognition
Marcel Binz, Elif Akata, Matthias Bethge +37
Establishing a unified theory of cognition has been a major goal of psychology. While there have been previous attempts to instantiate such theories by building computational model…
Data Contamination Report from the 2024 CONDA Shared Task
Oscar Sainz, Iker García-Ferrero, Alon Jacovi +25
The 1st Workshop on Data Contamination (CONDA 2024) focuses on all relevant aspects of data contamination in natural language processing, where data contamination is understood as…
A Practitioner's Guide to Continual Multimodal Pretraining
Karsten Roth, Vishaal Udandarao, Sebastian Dziadzio +7
Multimodal foundation models serve numerous applications at the intersection of vision and language. Still, despite being pretrained on extensive data, they become outdated over ti…
CiteME: Can Language Models Accurately Cite Scientific Claims?
Ori Press, Andreas Hochlehnert, Ameya Prabhu +3
Thousands of new scientific papers are published each month. Such information overload complicates researcher efforts to stay current with the state-of-the-art as well as to verify…
No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance
Vishaal Udandarao, Ameya Prabhu, Adhiraj Ghosh +5
Web-crawled pretraining datasets underlie the impressive "zero-shot" evaluation performance of multimodal models, such as CLIP for classification/retrieval and Stable-Diffusion for…
Efficient Lifelong Model Evaluation in an Era of Rapid Progress
Ameya Prabhu, Vishaal Udandarao, Philip Torr +3
Standardized benchmarks drive progress in machine learning. However, with repeated testing, the risk of overfitting grows as algorithms over-exploit benchmark idiosyncrasies. In ou…
Visual Data-Type Understanding does not emerge from Scaling Vision-Language Models
Vishaal Udandarao, Max F. Burg, Samuel Albanie +1
Recent advances in the development of vision-language models (VLMs) are yielding remarkable success in recognizing visual semantic content, including impressive instances of compos…
SuS-X: Training-Free Name-Only Transfer of Vision-Language Models
Vishaal Udandarao, Ankush Gupta, Samuel Albanie
Contrastive Language-Image Pre-training (CLIP) has emerged as a simple yet effective way to train large-scale vision-language models. CLIP demonstrates impressive zero-shot classif…
It's LeVAsa not LevioSA! Latent Encodings for Valence-Arousal Structure Alignment
Surabhi S. Nath, Vishaal Udandarao, Jainendra Shukla
In recent years, great strides have been made in the field of affective computing. Several models have been developed to represent and quantify emotions. Two popular ones include (…
DisCont: Self-Supervised Visual Attribute Disentanglement using Context Vectors
Sarthak Bhagat, Vishaal Udandarao, Shagun Uppal
Disentangling the underlying feature attributes within an image with no prior supervision is a challenging task. Models that can disentangle attributes well provide greater interpr…
On the Inference of Soft Biometrics from Typing Patterns Collected in a Multi-device Environment
Vishaal Udandarao, Mohit Agrawal, Rajesh Kumar +1
In this paper, we study the inference of gender, major/minor (computer science, non-computer science), typing style, age, and height from the typing patterns collected from 117 ind…
COBRA: Contrastive Bi-Modal Representation Algorithm
Vishaal Udandarao, Abhishek Maiti, Deepak Srivatsav +3
There are a wide range of applications that involve multi-modal data, such as cross-modal retrieval, visual question-answering, and image captioning. Such applications are primaril…
Memeify: A Large-Scale Meme Generation System
Suryatej Reddy Vyalla, Vishaal Udandarao, Tanmoy Chakraborty
Interest in the research areas related to meme propagation and generation has been increasing rapidly in the last couple of years. Meme datasets available online are either specifi…
EDUQA: Educational Domain Question Answering System using Conceptual Network Mapping
Abhishek Agarwal, Nikhil Sachdeva, Raj Kamal Yadav +4
Most of the existing question answering models can be largely compiled into two categories: i) open domain question answering models that answer generic questions and use large-sca…