27 citations · 69 across the 16 of their papers we have counts for
16 papers
SHRED: Retain-Set-Free Unlearning via Self-Distillation with Logit Demotion
Zizhao Hu, Ameya Godbole, Johnny Tian-Zheng Wei +3
Machine unlearning for large language models (LLMs) aims to selectively remove memorized content such as private data, copyrighted text, or hazardous knowledge, without costly full…
Correcting test set contamination by spiking the training data
Johnny Tian-Zheng Wei, Jerry Li, Ameya Godbole +1
The literature on test set contamination largely focuses on detection, but the correction of contaminated test scores is underexplored. Our core proposal is to spike the training d…
Evaluating LLM-Simulated Conversations in Modeling Inconsistent and Uncollaborative Behaviors in Human Social Interaction
Ryo Kamoi, Ameya Godbole, Binglin Zhou +5
Simulating human conversations using large language models (LLMs) has emerged as a scalable methodology for modeling human social interaction. This paper reconsiders the evaluation…
Hubble: a Model Suite to Advance the Study of LLM Memorization
Johnny Tian-Zheng Wei, Ameya Godbole, Mohammad Aflah Khan +7
We present Hubble, a suite of fully open-source large language models (LLMs) for the scientific study of LLM memorization. Hubble models come in standard and perturbed variants: st…
TokenSmith: Streamlining Data Editing, Search, and Inspection for Large-Scale Language Model Training and Interpretability
Mohammad Aflah Khan, Ameya Godbole, Johnny Tian-Zheng Wei +5
Understanding the relationship between training data and model behavior during pretraining is crucial, but existing workflows make this process cumbersome, fragmented, and often in…
Interrogating LLM design under a fair learning doctrine
Johnny Tian-Zheng Wei, Maggie Wang, Ameya Godbole +2
The current discourse on large language models (LLMs) and copyright largely takes a "behavioral" perspective, focusing on model outputs and evaluating whether they are substantiall…
Verify with Caution: The Pitfalls of Relying on Imperfect Factuality Metrics
Ameya Godbole, Robin Jia
Improvements in large language models have led to increasing optimism that they can serve as reliable evaluators of natural language generation outputs. In this paper, we challenge…
Analysis of Plan-based Retrieval for Grounded Text Generation
Ameya Godbole, Nicholas Monath, Seungyeon Kim +3
In text generation, hallucinations refer to the generation of seemingly coherent text that contradicts established knowledge. One compelling hypothesis is that hallucinations occur…
SCENE: Self-Labeled Counterfactuals for Extrapolating to Negative Examples
Deqing Fu, Ameya Godbole, Robin Jia
Detecting negatives (such as non-entailment relationships, unanswerable questions, and false claims) is an important and challenging aspect of many natural language understanding t…
Benchmarking Long-tail Generalization with Likelihood Splits
Ameya Godbole, Robin Jia
In order to reliably process natural language, NLP systems must generalize to the long tail of rare utterances. We propose a method to create challenging benchmarks that require ge…
Knowledge Base Question Answering by Case-based Reasoning over Subgraphs
Rajarshi Das, Ameya Godbole, Ankita Naik +5
Question answering (QA) over knowledge bases (KBs) is challenging because of the diverse, essentially unbounded, types of reasoning patterns needed. However, we hypothesize in a la…
Case-based Reasoning for Natural Language Queries over Knowledge Bases
Rajarshi Das, Manzil Zaheer, Dung Thai +6
It is often challenging to solve a complex problem from scratch, but much easier if we can access other similar problems with their solutions -- a paradigm known as case-based reas…
Probabilistic Case-based Reasoning for Open-World Knowledge Graph Completion
Rajarshi Das, Ameya Godbole, Nicholas Monath +2
A case-based reasoning (CBR) system solves a new problem by retrieving `cases' that are similar to the given problem. If such a system can achieve high accuracy, it is appealing ow…
A Simple Approach to Case-Based Reasoning in Knowledge Bases
Rajarshi Das, Ameya Godbole, Shehzaad Dhuliawala +2
We present a surprisingly simple yet accurate approach to reasoning in knowledge graphs (KGs) that requires \emph{no training}, and is reminiscent of case-based reasoning in classi…
Multi-step Entity-centric Information Retrieval for Multi-Hop Question Answering
Ameya Godbole, Dilip Kavarthapu, Rajarshi Das +8
Multi-hop question answering (QA) requires an information retrieval (IR) system that can find \emph{multiple} supporting evidence needed to answer the question, making the retrieva…
Siamese Neural Networks with Random Forest for detecting duplicate question pairs
Ameya Godbole, Aman Dalmia, Sunil Kumar Sahu
Determining whether two given questions are semantically similar is a fairly challenging task given the different structures and forms that the questions can take. In this paper, w…