4 citations · 7 across the 16 of their papers we have counts for
17 papers
PERSONAWEAVER: Controllable Diversity Beyond Conventional Archetypes in Procedural Character Generation
Maan Qraitem, Kate Saenko, Bryan A. Plummer
Procedural character generation aims to populate games, simulations, and other virtual worlds with diverse characters. Large language models (LLMs) offer a promising foundation for…
Breaking the Assistant Mold: Modeling Behavioral Variation in LLM Based Procedural Character Generation
Maan Qraitem, Kate Saenko, Bryan A. Plummer
Procedural content generation has enabled vast virtual worlds through levels, maps, and quests, but large-scale character generation remains underexplored. We identify two alignmen…
Mull-Tokens: Modality-Agnostic Latent Thinking
Arijit Ray, Ahmed Abdelkader, Chengzhi Mao +5
Reasoning goes beyond language; the real world requires reasoning about space, time, affordances, and much more that words alone cannot convey. Existing multimodal models exploring…
BabyVLM-V2: Toward Developmentally Grounded Pretraining and Benchmarking of Vision Foundation Models
Shengao Wang, Wenqi Wang, Zecheng Wang +20
Early children's developmental trajectories set up a natural goal for sample-efficient pretraining of vision foundation models. We introduce BabyVLM-V2, a developmentally grounded…
Scaling Up Temporal Domain Generalization via Temporal Experts Averaging
Aoming Liu, Kevin Miller, Venkatesh Saligrama +4
Temporal Domain Generalization (TDG) aims to generalize across temporal distribution shifts, e.g., lexical change over time. Prior work often addresses this by predicting future mo…
Web Artifact Attacks Disrupt Vision Language Models
Maan Qraitem, Piotr Teterwak, Kate Saenko +1
Vision-language models (VLMs) (e.g. CLIP, LLaVA) are trained on large-scale, lightly curated web datasets, leading them to learn unintended correlations between semantic concepts a…
OP-LoRA: The Blessing of Dimensionality
Piotr Teterwak, Kate Saenko, Bryan A. Plummer +1
Low-rank adapters (LoRA) enable finetuning of large models with only a small number of parameters. However, they often suffer from an ill-conditioned loss landscape, leading to dif…
SAT: Dynamic Spatial Aptitude Training for Multimodal Language Models
Arijit Ray, Jiafei Duan, Ellis Brown +9
Reasoning about motion and space is a fundamental cognitive capability that is required by multiple real-world applications. While many studies highlight that large multimodal lang…
Is Large-Scale Pretraining the Secret to Good Domain Generalization?
Piotr Teterwak, Kuniaki Saito, Theodoros Tsiligkaridis +2
Multi-Source Domain Generalization (DG) is the task of training on multiple source domains and achieving high classification performance on unseen target domains. Recent methods co…
KiVA: Kid-inspired Visual Analogies for Testing Large Multimodal Models
Eunice Yiu, Maan Qraitem, Anisa Noor Majhi +5
This paper investigates visual analogical reasoning in large multimodal models (LMMs) compared to human adults and children. A "visual analogy" is an abstract rule inferred from on…
SLANT: Spurious Logo ANalysis Toolkit
Maan Qraitem, Piotr Teterwak, Kate Saenko +1
Online content is filled with logos, from ads and social media posts to website branding and product placements. Consequently, these logos are prevalent in the extensive web-scrape…
Tell Me What's Next: Textual Foresight for Generic UI Representations
Andrea Burns, Kate Saenko, Bryan A. Plummer
Mobile app user interfaces (UIs) are rich with action, text, structure, and image content that can be utilized to learn generic UI representations for tasks like automating user co…
Koala: Key frame-conditioned long video-LLM
Reuben Tan, Ximeng Sun, Ping Hu +5
Long video question answering is a challenging task that involves recognizing short-term activities and reasoning about their fine-grained relationships. State-of-the-art video Lar…
Vision-LLMs Can Fool Themselves with Self-Generated Typographic Attacks
Maan Qraitem, Nazia Tasnim, Piotr Teterwak +2
Typographic attacks, adding misleading text to images, can deceive vision-language models (LVLMs). The susceptibility of recent large LVLMs like GPT4-V to such attacks is understud…
Learning to Compose SuperWeights for Neural Parameter Allocation Search
Piotr Teterwak, Soren Nelson, Nikoli Dryden +3
Neural parameter allocation search (NPAS) automates parameter sharing by obtaining weights for a network given an arbitrary, fixed parameter budget. Prior work has two major drawba…
CLAMP: Contrastive LAnguage Model Prompt-tuning
Piotr Teterwak, Ximeng Sun, Bryan A. Plummer +2
Large language models (LLMs) have emerged as powerful general-purpose interfaces for many machine learning problems. Recent work has adapted LLMs to generative visual tasks like im…
Lasagna: Layered Score Distillation for Disentangled Object Relighting
Dina Bashkirova, Arijit Ray, Rupayan Mallick +4
Professional artists, photographers, and other visual content creators use object relighting to establish their photo's desired effect. Unfortunately, manual tools that allow relig…