37 citations · 113 across the 18 of their papers we have counts for
18 papers
Multimodal Unlearning Across Vision, Language, Video, and Audio: Survey of Methods, Datasets, and Benchmarks
Nobin Sarwar, Shubhashis Roy Dipta, Zheyuan Liu +1
With the growing adoption of VLMs, DMs, LLMs, and AFMs, these multimodal foundation models can inadvertently encode sensitive, copyrighted, biased, or unsafe cross-modal associatio…
Identifying and Evaluating Inactive Heads in Pretrained LLMs
Pedro Sandoval-Segura, Xijun Wang, Ashwinee Panda +4
Attention is foundational to large language models (LLMs), enabling different heads to have diverse focus on relevant input tokens. However, learned behaviors like attention sinks,…
Refusal Tokens: A Simple Way to Calibrate Refusals in Large Language Models
Neel Jain, Aditya Shrivastava, Chenyang Zhu +6
A key component of building safe and reliable language models is enabling the models to appropriately refuse to follow certain instructions or answer certain questions. We may want…
Be like a Goldfish, Don't Memorize! Mitigating Memorization in Generative LLMs
Abhimanyu Hans, Yuxin Wen, Neel Jain +8
Large language models can memorize and repeat their training data, causing privacy and copyright risks. To mitigate memorization, we introduce a subtle modification to the next-tok…
LMD3: Language Model Data Density Dependence
John Kirchenbauer, Garrett Honke, Gowthami Somepalli +5
We develop a methodology for analyzing language model task performance at the individual example level based on training data density estimation. Experiments with paraphrasing as a…
CinePile: A Long Video Question Answering Dataset and Benchmark
Ruchit Rawal, Khalid Saifullah, Miquel Farré +4
Current datasets for long-form video understanding often fall short of providing genuine long-form comprehension challenges, as many tasks derived from these datasets can be succes…
Measuring Style Similarity in Diffusion Models
Gowthami Somepalli, Anubhav Gupta, Kamal Gupta +5
Generative models are now widely used by graphic designers and artists. Prior works have shown that these models remember and often replicate content from their training data durin…
Privacy Backdoors: Enhancing Membership Inference through Poisoning Pre-trained Models
Yuxin Wen, Leo Marchyok, Sanghyun Hong +3
It is commonplace to produce application-specific models by fine-tuning large pre-trained models using a small bespoke dataset. The widespread availability of foundation model chec…
Generating Potent Poisons and Backdoors from Scratch with Guided Diffusion
Hossein Souri, Arpit Bansal, Hamid Kazemi +7
Modern neural networks are often trained on massive datasets that are web scraped with minimal human inspection. As a result of this insecure curation pipeline, an adversary can po…
What do we learn from inverting CLIP models?
Hamid Kazemi, Atoosa Chegini, Jonas Geiping +2
We employ an inversion-based approach to examine CLIP models. Our examination reveals that inverting CLIP models results in the generation of images that exhibit semantic alignment…
Spotting LLMs With Binoculars: Zero-Shot Detection of Machine-Generated Text
Abhimanyu Hans, Avi Schwarzschild, Valeriia Cherepanova +5
Detecting text generated by modern large language models is thought to be hard, as both LLMs and humans can exhibit a wide range of complex behaviors. However, we find that a score…
Perspectives on the State and Future of Deep Learning - 2023
Micah Goldblum, Anima Anandkumar, Richard Baraniuk +7
The goal of this series is to chronicle opinions and issues in the field of machine learning as they stand today and as they change over time. The plan is to host this survey perio…
Object Recognition as Next Token Prediction
Kaiyu Yue, Bor-Chun Chen, Jonas Geiping +3
We present an approach to pose object recognition as next token prediction. The idea is to apply a language decoder that auto-regressively predicts the text tokens from image embed…
Battle of the Backbones: A Large-Scale Comparison of Pretrained Models across Computer Vision Tasks
Micah Goldblum, Hossein Souri, Renkun Ni +10
Neural network based computer vision systems are typically built on a backbone, a pretrained or randomly initialized feature extractor. Several years ago, the default option was an…
A Performance-Driven Benchmark for Feature Selection in Tabular Deep Learning
Valeriia Cherepanova, Roman Levin, Gowthami Somepalli +5
Academic tabular benchmarks often contain small sets of curated features. In contrast, data scientists typically collect as many features as possible into their datasets, and even…
A Simple and Efficient Baseline for Data Attribution on Images
Vasu Singla, Pedro Sandoval-Segura, Micah Goldblum +2
Data attribution methods play a crucial role in understanding machine learning models, providing insight into which training data points are most responsible for model outputs duri…
NEFTune: Noisy Embeddings Improve Instruction Finetuning
Neel Jain, Ping-yeh Chiang, Yuxin Wen +10
We show that language model finetuning can be improved, sometimes dramatically, with a simple augmentation. NEFTune adds noise to the embedding vectors during training. Standard fi…
Baseline Defenses for Adversarial Attacks Against Aligned Language Models
Neel Jain, Avi Schwarzschild, Yuxin Wen +7
As Large Language Models quickly become ubiquitous, it becomes critical to understand their security vulnerabilities. Recent work shows that text optimizers can produce jailbreakin…