45 citations · 135 across the 16 of their papers we have counts for
21 papers
All modalities are equal, but video is more equal: Closing the Cross-Attention Gap in Joint Video Generation
Ohad Rahamim, Dvir Samuel, Idan Schwartz +1
Video is a rich representation of a physical event, capturing appearance, geometry, motion, and temporal evolution. Other modalities, such as 3D body motion or audio, encode narrow…
TempoControl: Temporal Attention Guidance for Text-to-Video Models
Shira Schiber, Ofir Lindenbaum, Idan Schwartz
Recent advances in generative video models have enabled the creation of high-quality videos based on natural language prompts. However, these models frequently lack fine-grained te…
Single Image Iterative Subject-driven Generation and Editing
Yair Shpitzer, Gal Chechik, Idan Schwartz
Personalizing image generation and editing is particularly challenging when we only have a few images of the subject, or even a single image. A common approach to personalization i…
Detection-Driven Object Count Optimization for Text-to-Image Diffusion Models
Oz Zafar, Yuval Cohen, Lior Wolf +1
Accurately controlling object count in text-to-image generation remains a key challenge. Supervised methods often fail, as training data rarely covers all count variations. Methods…
LaMI: Augmenting Large Language Models via Late Multi-Image Fusion
Guy Yariv, Idan Schwartz, Yossi Adi +1
Commonsense reasoning often requires both textual and visual knowledge, yet Large Language Models (LLMs) trained solely on text lack visual grounding (e.g., "what color is an emper…
Diverse and Aligned Audio-to-Video Generation via Text-to-Video Model Adaptation
Guy Yariv, Itai Gat, Sagie Benaim +3
We consider the task of generating diverse and realistic videos guided by natural audio samples from a wide variety of semantic classes. For this task, the videos are required to b…
AudioToken: Adaptation of Text-Conditioned Diffusion Models for Audio-to-Image Generation
Guy Yariv, Itai Gat, Lior Wolf +2
In recent years, image generation has shown a great leap in performance, where diffusion models play a central role. Although generating high-quality images, such models are mainly…
Discriminative Class Tokens for Text-to-Image Diffusion Models
Idan Schwartz, Vésteinn Snæbjarnarson, Hila Chefer +4
Recent advances in text-to-image diffusion models have enabled the generation of diverse and high-quality images. While impressive, the images often fall short of depicting subtle…
Describing Sets of Images with Textual-PCA
Oded Hupert, Idan Schwartz, Lior Wolf
We seek to semantically describe a set of images, capturing both the attributes of single images and the variations within the set. Our procedure is analogous to Principle Componen…
Zero-Shot Video Captioning with Evolving Pseudo-Tokens
Yoad Tewel, Yoav Shalev, Roy Nadler +2
We introduce a zero-shot video captioning method that employs two frozen networks: the GPT-2 language model and the CLIP image-text matching model. The matching score is used to st…
Optimizing Relevance Maps of Vision Transformers Improves Robustness
Hila Chefer, Idan Schwartz, Lior Wolf
It has been observed that visual classification models often rely mostly on the image background, neglecting the foreground, which hurts their robustness to distribution changes. T…
Latent Space Explanation by Intervention
Itai Gat, Guy Lorberbom, Idan Schwartz +1
The success of deep neural nets heavily relies on their ability to encode complex relations between their input and their output. While this property serves to fit the training dat…
ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmetic
Yoad Tewel, Yoav Shalev, Idan Schwartz +1
Recent text-to-image matching models apply contrastive learning to large corpora of uncurated pairs of images and sentences. While such models can provide a powerful score for matc…
Perceptual Score: What Data Modalities Does Your Model Perceive?
Itai Gat, Idan Schwartz, Alexander Schwing
Machine learning advances in the last decade have relied significantly on large-scale datasets that continue to grow in size. Increasingly, those datasets also contain different da…
Video and Text Matching with Conditioned Embeddings
Ameen Ali, Idan Schwartz, Tamir Hazan +1
We present a method for matching a text sentence from a given corpus to a given video clip and vice versa. Traditionally video and text matching is done by learning a shared embedd…
Ordered Attention for Coherent Visual Storytelling
Tom Braude, Idan Schwartz, Alexander Schwing +1
We address the problem of visual storytelling, i.e., generating a story for a given sequence of images. While each sentence of the story should describe a corresponding image, a co…
Ensemble of MRR and NDCG models for Visual Dialog
Idan Schwartz
Assessing an AI agent that can converse in human language and understand visual content is challenging. Generation metrics, such as BLEU scores favor correct syntax over semantics.…
Removing Bias in Multi-modal Classifiers: Regularization by Maximizing Functional Entropies
Itai Gat, Idan Schwartz, Alexander Schwing +1
Many recent datasets contain a variety of different data modalities, for instance, image, question, and answer data in visual question answering (VQA). When training deep net class…
A Simple Baseline for Audio-Visual Scene-Aware Dialog
Idan Schwartz, Alexander Schwing, Tamir Hazan
The recently proposed audio-visual scene-aware dialog task paves the way to a more data-driven way of learning virtual assistants, smart speakers and car navigation systems. Howeve…
Factor Graph Attention
Idan Schwartz, Seunghak Yu, Tamir Hazan +1
Dialog is an effective way to exchange information, but subtle details and nuances are extremely important. While significant progress has paved a path to address visual dialog wit…
High-Order Attention Models for Visual Question Answering
Idan Schwartz, Alexander G. Schwing, Tamir Hazan
The quest for algorithms that enable cognitive abilities is an important part of machine learning. A common trait in many recently investigated cognitive-like tasks is that they ta…