58 citations · 407 across the 33 of their papers we have counts for
24 papers · 1 filter
EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the Backbone
Shraman Pramanick, Yale Song, Sayan Nag +5
Video-language pre-training (VLP) has become increasingly important due to its ability to generalize to various vision and language tasks. However, existing egocentric VLP framewor…
Egocentric Video Task Translation @ Ego4D Challenge 2022
Zihui Xue, Yale Song, Kristen Grauman +1
This technical report describes the EgoTask Translation approach that explores relations among a set of egocentric video tasks in the Ego4D challenge. To improve the primary task o…
Egocentric Video Task Translation
Zihui Xue, Yale Song, Kristen Grauman +1
Different video understanding tasks are typically treated in isolation, and even with distinct types of curated data (e.g., classifying sports in one dataset, tracking animals in a…
PatchBlender: A Motion Prior for Video Transformers
Gabriele Prato, Yale Song, Janarthanan Rajendran +3
Transformers have become one of the dominant architectures in the field of computer vision. However, there are yet several challenges when applying such architectures to video data…
Video Summarization Overview
Mayu Otani, Yale Song, Yang Wang
With the broad growth of video capturing devices and applications on the web, it is more demanding to provide desired video content for users efficiently. Video summarization facil…
One Network Doesn't Rule Them All: Moving Beyond Handcrafted Architectures in Self-Supervised Learning
Sharath Girish, Debadeepta Dey, Neel Joshi +5
The current literature on self-supervised learning (SSL) focuses on developing learning objectives to train neural networks more effectively on unlabeled data. The typical developm…
DOC2PPT: Automatic Presentation Slides Generation from Scientific Documents
Tsu-Jui Fu, William Yang Wang, Daniel McDuff +1
Creating presentation materials requires complex multimodal reasoning skills to summarize key concepts and arrange them in a logical and visually pleasing manner. Can machines lear…
ACAV100M: Automatic Curation of Large-Scale Datasets for Audio-Visual Video Representation Learning
Sangho Lee, Jiwan Chung, Youngjae Yu +4
The natural association between visual observations and their corresponding sound provides powerful self-supervisory signals for learning video representations, which makes the eve…
Learning to Transfer Visual Effects from Videos to Images
Christopher Thomas, Yale Song, Adriana Kovashka
We study the problem of animating images by transferring spatio-temporal visual effects (such as melting) from a collection of videos. We tackle two primary challenges in visual ef…
Parameter Efficient Multimodal Transformers for Video Representation Learning
Sangho Lee, Youngjae Yu, Gunhee Kim +3
The recent success of Transformers in the language domain has motivated adapting it to a multimodal setting, where a new visual model is trained in tandem with an already pretraine…
Unpaired Image-to-Speech Synthesis with Multimodal Information Bottleneck
Shuang Ma, Daniel McDuff, Yale Song
Deep generative models have led to significant advances in cross-modal generation such as text-to-image synthesis. Training these models typically requires paired data with direct…
Image to Video Domain Adaptation Using Web Supervision
Andrew Kae, Yale Song
Training deep neural networks typically requires large amounts of labeled data which may be scarce or expensive to obtain for a particular target domain. As an alternative, we can…
Polysemous Visual-Semantic Embedding for Cross-Modal Retrieval
Yale Song, Mohammad Soleymani
Visual-semantic embedding aims to find a shared latent space where related visual and textual instances are close to each other. Most current methods learn injective embedding func…
M3D-GAN: Multi-Modal Multi-Domain Translation with Universal Attention
Shuang Ma, Daniel McDuff, Yale Song
Generative adversarial networks have led to significant advances in cross-modal/domain translation. However, typically these networks are designed for a specific task (e.g., dialog…
Characterizing Bias in Classifiers using Generative Models
Daniel McDuff, Shuang Ma, Yale Song +1
Models that are learned from real-world data are often biased because the data used to train them is biased. This can propagate systemic human biases that exist and ultimately lead…
Video Prediction with Appearance and Motion Conditions
Yunseok Jang, Gunhee Kim, Yale Song
Video prediction aims to generate realistic future frames by learning dynamic visual patterns. One fundamental challenge is to deal with future uncertainty: How should a model beha…
Cross-Modal Retrieval with Implicit Concept Association
Yale Song, Mohammad Soleymani
Traditional cross-modal retrieval assumes explicit association of concepts across modalities, where there is no ambiguity in how the concepts are linked to each other, e.g., when w…
Image2GIF: Generating Cinemagraphs using Recurrent Deep Q-Networks
Yipin Zhou, Yale Song, Tamara L. Berg
Given a still photograph, one can imagine how dynamic objects might move against a static background. This idea has been actualized in the form of cinemagraphs, where the motion of…
Improving Pairwise Ranking for Multi-label Image Classification
Yuncheng Li, Yale Song, Jiebo Luo
Learning to rank has recently emerged as an attractive technique to train deep convolutional neural networks for various computer vision tasks. Pairwise ranking, in particular, has…
Learning from Noisy Labels with Distillation
Yuncheng Li, Jianchao Yang, Yale Song +3
The ability of learning from noisy labels is very useful in many visual recognition tasks, as a vast amount of data with noisy labels are relatively easy to obtain. Traditionally,…
TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering
Yunseok Jang, Yale Song, Youngjae Yu +2
Vision and language understanding has emerged as a subject undergoing intense study in Artificial Intelligence. Among many tasks in this line of research, visual question answering…
Real-Time Video Highlights for Yahoo Esports
Yale Song
Esports has gained global popularity in recent years and several companies have started offering live streaming videos of esports games and events. This creates opportunities to de…
Video2GIF: Automatic Generation of Animated GIFs from Video
Michael Gygli, Yale Song, Liangliang Cao
We introduce the novel problem of automatically generating animated GIFs from video. GIFs are short looping video with no sound, and a perfect combination between image and video t…
TGIF: A New Dataset and Benchmark on Animated GIF Description
Yuncheng Li, Yale Song, Liangliang Cao +4
With the recent popularity of animated GIFs on social media, there is need for ways to index them with rich metadata. To advance research on animated GIF understanding, we collecte…