12 papers
Input- and Feature-Space Correction for Cross-Hospital IHC Grading
Jongwoon Lee, Jhair Gallardo, Christopher Kanan
Pathology foundation models (FMs) are trained predominantly on hematoxylin and eosin (H&E) images but are increasingly used as frozen feature extractors for immunohistochemistry (I…
LayerRoPE: Dynamic Depth-wise Magnitude & Angular Superposition
Shikhar Srivastava, Christopher Kanan
As data propagates through a Transformer, the norm of its hidden states grows by orders of magnitude with depth, a phenomenon framed as 'curse of depth' and nearly universally trea…
Lost or Hidden? A Concept-Level Forgetting in Supervised Continual Learning
Katarzyna Filus, Kamil Faber, Roberto Corizzo +1
Continual learning studies how models can adapt to new tasks while retaining previously acquired knowledge. Although a broad spectrum of methods has been proposed to mitigate catas…
Unlocking ImageNet's Multi-Object Nature: Automated Large-Scale Multilabel Annotation
Junyu Chen, Md Yousuf Harun, Christopher Kanan
The original ImageNet benchmark enforces a single-label assumption, despite many images depicting multiple objects. This leads to label noise and limits the richness of the learnin…
A Good Start Matters: Enhancing Continual Learning with Data-Driven Weight Initialization
Md Yousuf Harun, Christopher Kanan
To adapt to real-world data streams, continual learning (CL) systems must rapidly learn new concepts while preserving and utilizing prior knowledge. When it comes to adding new inf…
Controlling Neural Collapse Enhances Out-of-Distribution Detection and Transfer Learning
Md Yousuf Harun, Jhair Gallardo, Christopher Kanan
Out-of-distribution (OOD) detection and OOD generalization are widely studied in Deep Neural Networks (DNNs), yet their relationship remains poorly understood. We empirically show…
INSIGHT: Explainable Weakly-Supervised Medical Image Analysis
Wenbo Zhang, Junyu Chen, Christopher Kanan
Due to their large sizes, volumetric scans and whole-slide pathology images (WSIs) are often processed by extracting embeddings from local regions and then an aggregator makes pred…
Improving Multimodal Large Language Models Using Continual Learning
Shikhar Srivastava, Md Yousuf Harun, Robik Shrestha +1
Generative large language models (LLMs) exhibit impressive capabilities, which can be further augmented by integrating a pre-trained vision model into the original LLM to create a…
Revisiting Multi-Modal LLM Evaluation
Jian Lu, Shikhar Srivastava, Junyu Chen +4
With the advent of multi-modal large language models (MLLMs), datasets used for visual question answering (VQA) and referring expression comprehension have seen a resurgence. Howev…
What Variables Affect Out-of-Distribution Generalization in Pretrained Models?
Md Yousuf Harun, Kyungbok Lee, Jhair Gallardo +2
Embeddings produced by pre-trained deep neural networks (DNNs) are widely used; however, their efficacy for downstream tasks can vary widely. We study the factors influencing trans…
DMC4ML: Data Movement Complexity for Machine Learning
Chen Ding, Christopher Kanan, Dylan McKellips +3
The greatest demand for today's computing is machine learning. This paper analyzes three machine learning algorithms: transformers, spatial convolution, and FFT. The analysis is no…
BloomVQA: Assessing Hierarchical Multi-modal Comprehension
Yunye Gong, Robik Shrestha, Jared Claypoole +4
We propose a novel VQA dataset, BloomVQA, to facilitate comprehensive evaluation of large vision-language models on comprehension tasks. Unlike current benchmarks that often focus…