607 citations · 1.4k across the 40 of their papers we have counts for
45 papers
IsoVRBench: Video Reward Modeling Without Human Feedback
Shuyue Stella Li, Jacqueline He, Jiahui Chen +5
Video reward models are trained and evaluated on human preferences between videos that often differ simultaneously on properties such as prompt alignment, visual fidelity, motion,…
Gacha Decoding: Eliciting Diverse Generations Through Instruction Following
Scott Geng, Yufei Zhang, Joseph Lee +3
We introduce Gacha Decoding, an inference-time method for eliciting diverse language model generations that scales with model capability. Across open-ended domains (in-the-wild cha…
Reasoning over mathematical objects: on-policy reward modeling and test time aggregation
Pranjal Aggarwal, Marjan Ghazvininejad, Seungone Kim +18
The ability to precisely derive mathematical objects is a core requirement for downstream STEM applications, including mathematics, physics, and chemistry, where reasoning must cul…
Beyond Language Modeling: An Exploration of Multimodal Pretraining
Shengbang Tong, David Fan, John Nguyen +18
The visual world offers a critical axis for advancing foundation models beyond language. Despite growing interest in this direction, the design space for native multimodal models r…
Unified Text-Image Generation with Weakness-Targeted Post-Training
Jiahui Chen, Philippe Hansen-Estruch, Xiaochuang Han +7
Unified multimodal generation architectures that jointly produce text and images have recently emerged as a promising direction for text-to-image (T2I) synthesis. However, many exi…
Multimodal RewardBench 2: Evaluating Omni Reward Models for Interleaved Text and Image
Yushi Hu, Reyhane Askari-Hemmat, Melissa Hall +3
Reward models (RMs) are essential for training large language models (LLMs), but remain underexplored for omni models that handle interleaved image and text sequences. We introduce…
GenEval 2: Addressing Benchmark Drift in Text-to-Image Evaluation
Amita Kamath, Kai-Wei Chang, Ranjay Krishna +3
Automating Text-to-Image (T2I) model evaluation is challenging; a judge model must be used to score correctness, and test prompts must be selected to be challenging for current T2I…
TV2TV: A Unified Framework for Interleaved Language and Video Generation
Xiaochuang Han, Youssef Emad, Melissa Hall +15
Video generation models are rapidly advancing, but can still struggle with complex video outputs that require significant semantic branching or repeated high-level reasoning about…
Self-Improving VLM Judges Without Human Annotations
Inna Wanyin Lin, Yushi Hu, Shuyue Stella Li +5
Effective judges of Vision-Language Models (VLMs) are crucial for model development. Current methods for training VLM judges mainly rely on large-scale human preference annotations…
Improving Chain-of-Thought Efficiency for Autoregressive Image Generation
Zeqi Gu, Markos Georgopoulos, Xiaoliang Dai +10
Autoregressive multimodal large language models have recently gained popularity for image generation, driven by advances in foundation models. To enhance alignment and detail, newe…
reWordBench: Benchmarking and Improving the Robustness of Reward Models with Transformed Inputs
Zhaofeng Wu, Michihiro Yasunaga, Andrew Cohen +3
Reward models have become a staple in modern NLP, serving as not only a scalable text evaluator, but also an indispensable component in many alignment recipes and inference-time al…
Multimodal RewardBench: Holistic Evaluation of Reward Models for Vision Language Models
Michihiro Yasunaga, Luke Zettlemoyer, Marjan Ghazvininejad
Reward models play an essential role in training vision-language models (VLMs) by assessing output quality to enable aligning with human preferences. Despite their importance, the…
Learning to Plan & Reason for Evaluation with Thinking-LLM-as-a-Judge
Swarnadeep Saha, Xian Li, Marjan Ghazvininejad +2
LLM-as-a-Judge models generate chain-of-thought (CoT) sequences intended to capture the step-bystep reasoning process that underlies the final evaluation of a response. However, du…
ALMA: Alignment with Minimal Annotation
Michihiro Yasunaga, Leonid Shamis, Chunting Zhou +4
Recent approaches to large language model (LLM) alignment typically require millions of human annotations or rely on external aligned models for synthetic data generation. This pap…
JPEG-LM: LLMs as Image Generators with Canonical Codec Representations
Xiaochuang Han, Marjan Ghazvininejad, Pang Wei Koh +1
Recent work in image and video generation has been adopting the autoregressive LLM architecture due to its generality and potentially easy integration into multi-modal systems. The…
GenEARL: A Training-Free Generative Framework for Multimodal Event Argument Role Labeling
Hritik Bansal, Po-Nien Kung, P. Jeffrey Brantingham +2
Multimodal event argument role labeling (EARL), a task that assigns a role for each event participant (object) in an image is a complex challenge. It requires reasoning over the en…
BLESS: Benchmarking Large Language Models on Sentence Simplification
Tannon Kew, Alison Chi, Laura Vásquez-Rodríguez +4
We present BLESS, a comprehensive performance benchmark of the most recent state-of-the-art large language models (LLMs) on the task of text simplification (TS). We examine how wel…
David helps Goliath: Inference-Time Collaboration Between Small Specialized and Large General Diffusion LMs
Xiaochuang Han, Sachin Kumar, Yulia Tsvetkov +1
Diffusion-based language models are emerging as a promising alternative to autoregressive LMs: they approach the competence of autoregressive LMs while offering nuanced controllabi…
Dictionary-based Phrase-level Prompting of Large Language Models for Machine Translation
Marjan Ghazvininejad, Hila Gonen, Luke Zettlemoyer
Large language models (LLMs) demonstrate remarkable machine translation (MT) abilities via prompting, even though they were not explicitly trained for this task. However, even give…
Representation Deficiency in Masked Language Modeling
Yu Meng, Jitin Krishnan, Sinong Wang +6
Masked Language Modeling (MLM) has been one of the most prominent approaches for pretraining bidirectional text encoders due to its simplicity and effectiveness. One notable concer…
XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Models
Davis Liang, Hila Gonen, Yuning Mao +5
Large multilingual language models typically rely on a single vocabulary shared across 100+ languages. As these models have increased in parameter count and depth, vocabulary size…
In-context Examples Selection for Machine Translation
Sweta Agrawal, Chunting Zhou, Mike Lewis +2
Large-scale generative models show an impressive ability to perform a wide range of Natural Language Processing (NLP) tasks using in-context learning, where a few examples are used…
A Review on Language Models as Knowledge Bases
Badr AlKhamissi, Millicent Li, Asli Celikyilmaz +2
Recently, there has been a surge of interest in the NLP community on the use of pretrained Language Models (LMs) as Knowledge Bases (KBs). Researchers have shown that LMs trained o…
Natural Language to Code Translation with Execution
Freda Shi, Daniel Fried, Marjan Ghazvininejad +2
Generative models of code, pretrained on large corpora of programs, have shown great success in translating natural language to code (Chen et al., 2021; Austin et al., 2021; Li et…