7 papers
Muse Spark Safety & Preparedness Report
Cristina Menghini, Peter Ney, Hamza Kwisaba +117
Muse Spark is the latest large language model developed by Meta. In this report, we first present evaluations for catastrophic risk domains under Meta's Advanced AI Scaling Framewo…
Code World Model Preparedness Report
Daniel Song, Peter Ney, Cristina Menghini +21
This report documents the preparedness assessment of Code World Model (CWM), a model for code generation and reasoning about code from Meta. We conducted pre-release testing across…
Humanity's Last Exam
Long Phan, Alice Gatti, Ziwen Han +1144
Benchmarks are important tools for tracking the rapid advancements in large language model (LLM) capabilities. However, benchmarks are not keeping pace in difficulty: LLMs now achi…
Search-Time Data Contamination
Ziwen Han, Meher Mankikar, Julian Michael +1
Data contamination refers to the leakage of evaluation data into model training data, resulting in overfitting to supposedly held-out test sets and compromising test validity. We i…
Teaching LLMs How to Learn with Contextual Fine-Tuning
Younwoo Choi, Muhammad Adil Asif, Ziwen Han +2
Prompting Large Language Models (LLMs), or providing context on the expected model of operation, is an effective way to steer the outputs of such models to satisfy human desiderata…
Breach By A Thousand Leaks: Unsafe Information Leakage in `Safe' AI Responses
David Glukhov, Ziwen Han, Ilia Shumailov +2
Vulnerability of Frontier language models to misuse and jailbreaks has prompted the development of safety measures like filters and alignment training in an effort to ensure safety…