12 citations · 20 across the 23 of their papers we have counts for
11 papers · 1 filter
Probing the Representational Power of Sparse Autoencoders in Vision Models
Matthew Lyle Olson, Musashi Hinck, Neale Ratzlaff +4
Sparse Autoencoders (SAEs) have emerged as a popular tool for interpreting the hidden states of large language models (LLMs). By learning to reconstruct activations from a sparse b…
Analyzing Hierarchical Structure in Vision Models with Sparse Autoencoders
Matthew Lyle Olson, Musashi Hinck, Neale Ratzlaff +4
The ImageNet hierarchy provides a structured taxonomy of object categories, offering a valuable lens through which to analyze the representations learned by deep vision models. In…
LVLM-Compress-Bench: Benchmarking the Broader Impact of Large Vision-Language Model Compression
Souvik Kundu, Anahita Bhiwandiwalla, Sungduk Yu +6
Despite recent efforts in understanding the compression impact on large language models (LLMs) in terms of their downstream task performance and trustworthiness on relatively simpl…
Pruning the Paradox: How CLIP's Most Informative Heads Enhance Performance While Amplifying Bias
Avinash Madasu, Vasudev Lal, Phillip Howard
CLIP is one of the most popular foundation models and is heavily used for many vision-language tasks, yet little is known about its inner workings. As CLIP is increasingly deployed…
Training-Free Mitigation of Language Reasoning Degradation After Multimodal Instruction Tuning
Neale Ratzlaff, Man Luo, Xin Su +2
Multimodal models typically combine a powerful large language model (LLM) with a vision encoder and are then trained on multimodal data via instruction tuning. While this process a…
FiVL: A Framework for Improved Vision-Language Alignment through the Lens of Training, Evaluation and Explainability
Estelle Aflalo, Gabriela Ben Melech Stan, Tiep Le +5
Large Vision Language Models (LVLMs) have achieved significant progress in integrating visual and textual inputs for multimodal reasoning. However, a recurring challenge is ensurin…
Debias your Large Multi-Modal Model at Test-Time via Non-Contrastive Visual Attribute Steering
Neale Ratzlaff, Matthew Lyle Olson, Musashi Hinck +4
Large Multi-Modal Models (LMMs) have demonstrated impressive capabilities as general-purpose chatbots able to engage in conversations about visual inputs. However, their responses…
Debiasing Large Vision-Language Models by Ablating Protected Attribute Representations
Neale Ratzlaff, Matthew Lyle Olson, Musashi Hinck +3
Large Vision Language Models (LVLMs) such as LLaVA have demonstrated impressive capabilities as general-purpose chatbots that can engage in conversations about a provided input ima…
ClimDetect: A Benchmark Dataset for Climate Change Detection and Attribution
Sungduk Yu, Brian L. White, Anahita Bhiwandiwalla +6
Detecting and attributing temperature increases driven by climate change is crucial for understanding global warming and informing adaptation strategies. However, distinguishing hu…
LDM3D-VR: Latent Diffusion Model for 3D VR
Gabriela Ben Melech Stan, Diana Wofk, Estelle Aflalo +4
Latent diffusion models have proven to be state-of-the-art in the creation and manipulation of visual outputs. However, as far as we know, the generation of depth maps jointly with…
LDM3D: Latent Diffusion Model for 3D
Gabriela Ben Melech Stan, Diana Wofk, Scottie Fox +8
This research paper proposes a Latent Diffusion Model for 3D (LDM3D) that generates both image and depth map data from a given text prompt, allowing users to generate RGBD images f…