6 papers
Multimodal Unlearning Across Vision, Language, Video, and Audio: Survey of Methods, Datasets, and Benchmarks
Nobin Sarwar, Shubhashis Roy Dipta, Zheyuan Liu +1
With the growing adoption of VLMs, DMs, LLMs, and AFMs, these multimodal foundation models can inadvertently encode sensitive, copyrighted, biased, or unsafe cross-modal associatio…
ExecTune: Effective Steering of Black-Box LLMs with Guide Models
Vijay Lingam, Aditya Golatkar, Anwesan Pal +6
For large language models deployed through black-box APIs, recurring inference costs often exceed one-time training costs. This motivates composed agentic systems that amortize exp…
Re-FORC: Adaptive Reward Prediction for Efficient Chain-of-Thought Reasoning
Renos Zabounidis, Aditya Golatkar, Michael Kleinman +3
We propose Re-FORC, an adaptive reward prediction method that, given a query, enables prediction of the expected future rewards as a function of the number of future thinking token…
Expansion Span: Combining Fading Memory and Retrieval in Hybrid State Space Models
Elvis Nunez, Luca Zancato, Benjamin Bowman +3
The "state" of State Space Models (SSMs) represents their memory, which fades exponentially over an unbounded span. By contrast, Attention-based models have "eidetic" (i.e., verbat…
PICASO: Permutation-Invariant Context Composition with State Space Models
Tian Yu Liu, Alessandro Achille, Matthew Trager +3
Providing Large Language Models with relevant contextual knowledge at inference time has been shown to greatly improve the quality of their generations. This is often achieved by p…
Training Data Protection with Compositional Diffusion Models
Aditya Golatkar, Alessandro Achille, Ashwin Swaminathan +1
We introduce Compartmentalized Diffusion Models (CDM), a method to train different diffusion models (or prompts) on distinct data sources and arbitrarily compose them at inference…