10 papers
FastGuide: Accelerating Reward Guidance for Diffusion Large Language Models
Darshan Thaker, Lachlan Ewen MacDonald, René Vidal
Gradient-based reward guidance provides a flexible way to use downstream reward models to control masked diffusion language models at inference time. However, its computational cos…
ANaLOG: Anisotropic Native-Latent Operator Guidance for Solving Inverse Problems
Darshan Thaker, Lachlan Ewen MacDonald, René Vidal
Native-latent guidance is a recent paradigm for solving inverse problems with latent diffusion models. It replaces repeated evaluations of the image-space forward model, each requi…
REALISTA: Realistic Latent Adversarial Attacks that Elicit LLM Hallucinations
Buyun Liang, Jinqi Luo, Liangzu Peng +6
Large language models (LLMs) achieve strong performance across many tasks but remain vulnerable to hallucinations, making it important to systematically evaluate their reliability…
SECA: Semantically Equivalent and Coherent Attacks for Eliciting LLM Hallucinations
Buyun Liang, Liangzu Peng, Jinqi Luo +3
Large Language Models (LLMs) are increasingly deployed in high-risk domains. However, state-of-the-art LLMs often exhibit hallucinations, raising serious concerns about their relia…
Conformal Information Pursuit for Interactively Guiding Large Language Models
Kwan Ho Ryan Chan, Yuyan Ge, Edgar Dobriban +2
A significant use case of instruction-finetuned Large Language Models (LLMs) is to solve question-answering tasks interactively. In this setting, an LLM agent is tasked with making…
KDA: A Knowledge-Distilled Attacker for Generating Diverse Prompts to Jailbreak LLMs
Buyun Liang, Kwan Ho Ryan Chan, Darshan Thaker +2
Jailbreak attacks exploit specific prompts to bypass LLM safeguards, causing the LLM to generate harmful, inappropriate, and misaligned content. Current jailbreaking methods rely h…
Frequency-Guided Posterior Sampling for Diffusion-Based Image Restoration
Darshan Thaker, Abhishek Goyal, René Vidal
Image restoration aims to recover high-quality images from degraded observations. When the degradation process is known, the recovery problem can be formulated as an inverse proble…
PaCE: Parsimonious Concept Engineering for Large Language Models
Jinqi Luo, Tianjiao Ding, Kwan Ho Ryan Chan +4
Large Language Models (LLMs) are being used for a wide variety of tasks. While they are capable of generating human-like responses, they can also produce undesirable output includi…
Learning Interpretable Queries for Explainable Image Classification with Information Pursuit
Stefan Kolek, Aditya Chattopadhyay, Kwan Ho Ryan Chan +3
Information Pursuit (IP) is an explainable prediction algorithm that greedily selects a sequence of interpretable queries about the data in order of information gain, updating its…
Contextual Knowledge Pursuit for Faithful Visual Synthesis
Jinqi Luo, Kwan Ho Ryan Chan, Dimitris Dimos +1
Modern text-to-vision generative models often hallucinate when the prompt describing the scene to be generated is underspecified. In large language models (LLMs), a prevalent strat…