50 citations · 113 across the 31 of their papers we have counts for
31 papers · 1 filter
Instruction-Based Video Editing by Repurposing an Image Editing Model
Yunpeng Bai, Yossi Gandelsman, Michaël Gharbi +1
Instruction-based video editing is commonly built on video-pretrained generative backbones: a video diffusion transformer is adapted, at considerable cost, to condition on a source…
Test-Time Training for Modality Order Consistency in Vision-Language Models
Aditi Gupta, Yossi Gandelsman
We find that vision-language models are sensitive to a specific semantically irrelevant change: the order in which the image and question are presented. Across three models and thr…
DuSPiT: Dual-Branch Sub-Patch Pixel Diffusion Transformer
Yunpeng Bai, Yossi Gandelsman, Michaël Gharbi
Diffusion Transformers achieve strong image generation performance, but most operate in compressed latent spaces. Pixel-space diffusion avoids this information loss, yet existing a…
Jailbreaking Vision-Language Models Through the Visual Modality
Aharon Azulay, Jan Dubiński, Zhuoyun Li +2
The visual modality of vision-language models (VLMs) is an underexplored attack surface for bypassing safety alignment. We introduce four jailbreak attacks exploiting the vision co…
The Unreasonable Effectiveness of Text Embedding Interpolation for Continuous Image Steering
Yigit Ekin, Yossi Gandelsman
We present a training-free framework for continuous and controllable image editing at test time for text-conditioned generative models. In contrast to prior approaches that rely on…
Interpreting ResNet-based CLIP via Neuron-Attention Decomposition
Edmund Bu, Yossi Gandelsman
We present a novel technique for interpreting the neurons in CLIP-ResNet by decomposing their contributions to the output into individual computation paths. More specifically, we a…
Vision Transformers Don't Need Trained Registers
Nick Jiang, Amil Dravid, Alexei Efros +1
We investigate the mechanism underlying a previously identified phenomenon in Vision Transformers - the emergence of high-norm tokens that lead to noisy attention maps (Darcet et a…
Teaching Humans Subtle Differences with DIFFusion
Mia Chiquier, Orr Avrech, Yossi Gandelsman +3
Scientific expertise often requires recognizing subtle visual differences that remain challenging to articulate even for domain experts. We present a system that leverages generati…
Steering CLIP's vision transformer with sparse autoencoders
Sonia Joseph, Praneet Suresh, Ethan Goldfarb +6
While vision models are highly capable, their internal mechanisms remain poorly understood -- a challenge which sparse autoencoders (SAEs) have helped address in language, but whic…
LLMs can see and hear without any training
Kumar Ashutosh, Yossi Gandelsman, Xinlei Chen +2
We present MILS: Multimodal Iterative LLM Solver, a surprisingly simple, training-free approach, to imbue multimodal capabilities into your favorite LLM. Leveraging their innate ab…
An Empirical Study of Autoregressive Pre-training from Videos
Jathushan Rajasegaran, Ilija Radosavovic, Rahul Ravishankar +3
We empirically study autoregressive pre-training from videos. To perform our study, we construct a series of autoregressive video models, called Toto. We treat videos as sequences…
Learning Video Representations without Natural Videos
Xueyang Yu, Xinlei Chen, Yossi Gandelsman
We show that useful video representations can be learned from synthetic videos and natural images, without incorporating natural videos in the training. We propose a progression of…
Interpreting and Editing Vision-Language Representations to Mitigate Hallucinations
Nick Jiang, Anish Kachinthaya, Suzie Petryk +1
We investigate the internal representations of vision-language models (VLMs) to address hallucinations, a persistent challenge despite advances in model size and training. We proje…
Quantifying and Enabling the Interpretability of CLIP-like Models
Avinash Madasu, Yossi Gandelsman, Vasudev Lal +1
CLIP is one of the most popular foundational models and is heavily used for many vision-language tasks. However, little is known about the inner workings of CLIP. To bridge this ga…
Interpreting the Weight Space of Customized Diffusion Models
Amil Dravid, Yossi Gandelsman, Kuan-Chieh Wang +4
We investigate the space of weights spanned by a large collection of customized diffusion models. We populate this space by creating a dataset of over 60,000 models, each of which…
Interpreting the Second-Order Effects of Neurons in CLIP
Yossi Gandelsman, Alexei A. Efros, Jacob Steinhardt
We interpret the function of individual neurons in CLIP by automatically describing them using text. Analyzing the direct effects (i.e. the flow from a neuron through the residual…
The More You See in 2D, the More You Perceive in 3D
Xinyang Han, Zelin Gao, Angjoo Kanazawa +2
Humans can infer 3D structure from 2D images of an object based on past experience and improve their 3D understanding as they see more images. Inspired by this behavior, we introdu…
Synthesizing Moving People with 3D Control
Boyi Li, Junming Chen, Jathushan Rajasegaran +3
In this paper, we present a diffusion model-based framework for animating people from a single image for a given target 3D motion sequence. Our approach has two core components: a)…
IMProv: Inpainting-based Multimodal Prompting for Computer Vision Tasks
Jiarui Xu, Yossi Gandelsman, Amir Bar +4
In-context learning allows adapting a model to new tasks given a task description at test time. In this paper, we present IMProv - a generative model that is able to in-context lea…
Idempotent Generative Network
Assaf Shocher, Amil Dravid, Yossi Gandelsman +3
We propose a new approach for generative modeling based on training a neural network to be idempotent. An idempotent operator is one that can be applied sequentially without changi…
Interpreting CLIP's Image Representation via Text-Based Decomposition
Yossi Gandelsman, Alexei A. Efros, Jacob Steinhardt
We investigate the CLIP image encoder by analyzing how individual model components affect the final representation. We decompose the image representation as a sum across individual…
Reconstruction of Patient-Specific Confounders in AI-based Radiologic Image Interpretation using Generative Pretraining
Tianyu Han, Laura Žigutytė, Luisa Huck +9
Detecting misleading patterns in automated diagnostic assistance systems, such as those powered by Artificial Intelligence, is critical to ensuring their reliability, particularly…
Test-Time Training on Video Streams
Renhao Wang, Yu Sun, Arnuv Tandon +4
Prior work has established Test-Time Training (TTT) as a general framework to further improve a trained model at test time. Before making a prediction on each test instance, the mo…
Rosetta Neurons: Mining the Common Units in a Model Zoo
Amil Dravid, Yossi Gandelsman, Alexei A. Efros +1
Do different neural networks, trained for various vision tasks, share some common representations? In this paper, we demonstrate the existence of common features we call "Rosetta N…