9 papers
Extracting memorized pieces of (copyrighted) books from open-weight language models
A. Feder Cooper, Mark A. Lemley, Allison Casasola +6
Plaintiffs and defendants in copyright lawsuits over generative AI often make sweeping, opposing claims about the extent to which large language models (LLMs) memorize protected ex…
Estimating near-verbatim extraction risk in language models with decoding-constrained beam search
A. Feder Cooper, Mark A. Lemley, Christopher De Sa +6
Recent work shows that standard greedy-decoding extraction methods for quantifying memorization in LLMs miss how extraction risk varies across sequences. Probabilistic extraction -…
Comparing AI Agents to Cybersecurity Professionals in Real-World Penetration Testing
Justin W. Lin, Eliot Krzysztof Jones, Donovan Julian Jasper +10
We present the first comprehensive evaluation of AI agents against human cybersecurity professionals in a live enterprise environment. We evaluate ten cybersecurity professionals a…
BountyBench: Dollar Impact of AI Agent Attackers and Defenders on Real-World Cybersecurity Systems
Andy K. Zhang, Joey Ji, Celeste Menders +31
AI agents have the potential to significantly alter the cybersecurity landscape. Here, we introduce the first framework to capture offensive and defensive cyber-capabilities in evo…
Audits Under Resource, Data, and Access Constraints: Scaling Laws For Less Discriminatory Alternatives
Sarah H. Cen, Salil Goyal, Zaynah Javed +3
AI audits play a critical role in AI accountability and safety. One branch of the law for which AI audits are particularly salient is anti-discrimination law. Several areas of anti…
Advancing Science- and Evidence-based AI Policy
Rishi Bommasani, Sanjeev Arora, Jennifer Chayes +17
AI policy should advance AI innovation by ensuring that its potential benefits are responsibly realized and widely shared. To achieve this, AI policymaking should place a premium o…