4 papers
Extracting books from production language models
Ahmed Ahmed, A. Feder Cooper, Sanmi Koyejo +1
Many unresolved legal questions over LLMs and copyright center on memorization: whether specific training data have been encoded in the model's weights during training, and whether…
SpecEval: Evaluating Model Adherence to Behavior Specifications
Ahmed Ahmed, Kevin Klyman, Yi Zeng +2
Companies that develop foundation models publish behavioral guidelines they pledge their models will follow, but it remains unclear if models actually do so. While providers such a…
AILuminate: Introducing v1.0 of the AI Risk and Reliability Benchmark from MLCommons
Shaona Ghosh, Heather Frase, Adina Williams +99
The rapid advancement and deployment of AI systems have created an urgent need for standard safety-evaluation frameworks. This paper introduces AILuminate v1.0, the first comprehen…
Independence Tests for Language Models
Sally Zhu, Ahmed Ahmed, Rohith Kuditipudi +1
We consider the following problem: given the weights of two models, can we test whether they were trained independently -- i.e., from independent random initializations? We conside…