24 citations · 50 across the 10 of their papers we have counts for
10 papers
Ctrl-Adapter: An Efficient and Versatile Framework for Adapting Diverse Controls to Any Diffusion Model
Han Lin, Jaemin Cho, Abhay Zala +1
ControlNets are widely used for adding spatial control to text-to-image diffusion models with different conditions, such as depth maps, scribbles/sketches, and human poses. However…
EnvGen: Generating and Adapting Environments via LLMs for Training Embodied Agents
Abhay Zala, Jaemin Cho, Han Lin +2
Recent SOTA approaches for embodied learning via interaction directly employ large language models (LLMs) as agents to determine the next steps in an environment. Due to their worl…
DiagrammerGPT: Generating Open-Domain, Open-Platform Diagrams via LLM Planning
Abhay Zala, Han Lin, Jaemin Cho +1
Text-to-image (T2I) generation has seen significant growth over the past few years. Despite this, there has been little work on generating diagrams with T2I models. A diagram is a…
VideoDirectorGPT: Consistent Multi-scene Video Generation via LLM-Guided Planning
Han Lin, Abhay Zala, Jaemin Cho +1
Recent text-to-video (T2V) generation methods have seen significant advancements. However, the majority of these works focus on producing short video clips of a single event (i.e.,…
Visual Programming for Text-to-Image Generation and Evaluation
Jaemin Cho, Abhay Zala, Mohit Bansal
As large language models have demonstrated impressive performance in many domains, recent works have adopted language models (LMs) as controllers of visual modules for vision-and-l…
Hierarchical Video-Moment Retrieval and Step-Captioning
Abhay Zala, Jaemin Cho, Satwik Kottur +4
There is growing interest in searching for information from large video corpora. Prior works have studied relevant tasks, such as text-based video retrieval, moment retrieval, vide…
CoSIm: Commonsense Reasoning for Counterfactual Scene Imagination
Hyounghun Kim, Abhay Zala, Mohit Bansal
As humans, we can modify our assumptions about a scene by imagining alternative objects or concepts in our minds. For example, we can easily anticipate the implications of the sun…
DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generation Models
Jaemin Cho, Abhay Zala, Mohit Bansal
Recently, DALL-E, a multimodal transformer language model, and its variants, including diffusion models, have shown high-quality text-to-image generation capabilities. However, des…
FixMyPose: Pose Correctional Captioning and Retrieval
Hyounghun Kim, Abhay Zala, Graham Burri +1
Interest in physical therapy and individual exercises such as yoga/dance has increased alongside the well-being trend. However, such exercises are hard to follow without expert gui…
ArraMon: A Joint Navigation-Assembly Instruction Interpretation Task in Dynamic Environments
Hyounghun Kim, Abhay Zala, Graham Burri +2
For embodied agents, navigation is an important ability but not an isolated goal. Agents are also expected to perform specific tasks after reaching the target location, such as pic…