193 citations · 327 across the 16 of their papers we have counts for
18 papers
FlexEControl: Flexible and Efficient Multimodal Control for Text-to-Image Generation
Xuehai He, Jian Zheng, Jacob Zhiyuan Fang +6
Controllable text-to-image (T2I) diffusion models generate images conditioned on both text prompts and semantic inputs of other modalities like edge maps. Nevertheless, current con…
Characterizing Video Question Answering with Sparsified Inputs
Shiyuan Huang, Robinson Piramuthu, Vicente Ordonez +2
In Video Question Answering, videos are often processed as a full-length sequence of frames to ensure minimal loss of information. Recent works have demonstrated evidence that spar…
Unsupervised Melody-Guided Lyrics Generation
Yufei Tian, Anjali Narayan-Chen, Shereen Oraby +7
Automatic song writing is a topic of significant practical interest. However, its research is largely hindered by the lack of training data due to copyright concerns and challenged…
Unsupervised Melody-to-Lyric Generation
Yufei Tian, Anjali Narayan-Chen, Shereen Oraby +8
Automatic melody-to-lyric generation is a task in which song lyrics are generated to go with a given melody. It is of significant practical interest and more challenging than uncon…
Decision Making for Human-in-the-loop Robotic Agents via Uncertainty-Aware Reinforcement Learning
Siddharth Singi, Zhanpeng He, Alvin Pan +5
In a Human-in-the-Loop paradigm, a robotic agent is able to act mostly autonomously in solving a task, but can request help from an external expert when needed. However, knowing wh…
RREx-BoT: Remote Referring Expressions with a Bag of Tricks
Gunnar A. Sigurdsson, Jesse Thomason, Gaurav S. Sukhatme +1
Household robots operate in the same space for years. Such robots incrementally build dynamic maps that can be used for tasks requiring remote object localization. However, benchma…
CLIP-Nav: Using CLIP for Zero-Shot Vision-and-Language Navigation
Vishnu Sashank Dorbala, Gunnar Sigurdsson, Robinson Piramuthu +2
Household environments are visually diverse. Embodied agents performing Vision-and-Language Navigation (VLN) in the wild must be able to handle this diversity, while also following…
Video in 10 Bits: Few-Bit VideoQA for Efficiency and Privacy
Shiyuan Huang, Robinson Piramuthu, Shih-Fu Chang +1
In Video Question Answering (VideoQA), answering general questions about a video requires its visual information. Yet, video often contains redundant information irrelevant to the…
A Simple Approach for Visual Rearrangement: 3D Mapping and Semantic Search
Brandon Trabucco, Gunnar Sigurdsson, Robinson Piramuthu +2
Physically rearranging objects is an important capability for embodied agents. Visual room rearrangement evaluates an agent's ability to rearrange objects in a room to a desired go…
Visual Grounding in Video for Unsupervised Word Translation
Gunnar A. Sigurdsson, Jean-Baptiste Alayrac, Aida Nematzadeh +5
There are thousands of actively spoken languages on Earth, but a single visual world. Grounding in this visual world has the potential to bridge the gap between all these languages…
Beyond the Camera: Neural Networks in World Coordinates
Gunnar A. Sigurdsson, Abhinav Gupta, Cordelia Schmid +1
Eye movement and strategic placement of the visual field onto the retina, gives animals increased resolution of the scene and suppresses distracting information. This fundamental s…
Charades-Ego: A Large-Scale Dataset of Paired Third and First Person Videos
Gunnar A. Sigurdsson, Abhinav Gupta, Cordelia Schmid +2
In Actor and Observer we introduced a dataset linking the first and third-person video understanding domains, the Charades-Ego Dataset. In this paper we describe the egocentric asp…
Actor and Observer: Joint Modeling of First and Third-Person Videos
Gunnar A. Sigurdsson, Abhinav Gupta, Cordelia Schmid +2
Several theories in cognitive neuroscience suggest that when people interact with the world, or simulate interactions, they do so from a first-person egocentric perspective, and se…
What Actions are Needed for Understanding Human Actions in Videos?
Gunnar A. Sigurdsson, Olga Russakovsky, Abhinav Gupta
What is the right way to reason about human activities? What directions forward are most promising? In this work, we analyze the current state of human activity understanding in vi…
Asynchronous Temporal Fields for Action Recognition
Gunnar A. Sigurdsson, Santosh Divvala, Ali Farhadi +1
Actions are more than just movements and trajectories: we cook to eat and we hold a cup to drink from it. A thorough understanding of videos requires going beyond appearance modeli…
Much Ado About Time: Exhaustive Annotation of Temporal Data
Gunnar A. Sigurdsson, Olga Russakovsky, Ali Farhadi +2
Large-scale annotated datasets allow AI systems to learn from and build upon the knowledge of the crowd. Many crowdsourcing techniques have been developed for collecting image anno…
Learning Visual Storylines with Skipping Recurrent Neural Networks
Gunnar A. Sigurdsson, Xinlei Chen, Abhinav Gupta
What does a typical visit to Paris look like? Do people first take photos of the Louvre and then the Eiffel Tower? Can we visually model a temporal event like "Paris Vacation" usin…
Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding
Gunnar A. Sigurdsson, Gül Varol, Xiaolong Wang +3
Computer vision has a great potential to help our daily lives by searching for lost keys, watering flowers or reminding us to take a pill. To succeed with such tasks, computer visi…