6 citations · 10 across the 9 of their papers we have counts for
9 papers
Muse Spark Safety & Preparedness Report
Cristina Menghini, Peter Ney, Hamza Kwisaba +117
Muse Spark is the latest large language model developed by Meta. In this report, we first present evaluations for catastrophic risk domains under Meta's Advanced AI Scaling Framewo…
Eval Factsheets: A Structured Framework for Documenting AI Evaluations
Florian Bordes, Candace Ross, Justine T Kao +2
The rapid proliferation of benchmarks has created significant challenges in reproducibility, transparency, and informed decision-making. However, unlike datasets and models -- whic…
What makes a good metric? Evaluating automatic metrics for text-to-image consistency
Candace Ross, Melissa Hall, Adriana Romero Soriano +1
Language models are increasingly being incorporated as components in larger AI systems for various purposes, from prompt optimization to automatic evaluation. In this work, we anal…
Improving Model Evaluation using SMART Filtering of Benchmark Datasets
Vipul Gupta, Candace Ross, David Pantoja +3
One of the most challenging problems facing NLP today is evaluation. Some of the most pressing issues pertain to benchmark saturation, data contamination, and diversity in the qual…
Decomposed evaluations of geographic disparities in text-to-image models
Abhishek Sureddy, Dishant Padalia, Nandhinee Periyakaruppa +6
Recent work has identified substantial disparities in generated images of different geographic regions, including stereotypical depictions of everyday objects like houses and cars.…
Changing Answer Order Can Decrease MMLU Accuracy
Vipul Gupta, David Pantoja, Candace Ross +2
As large language models (LLMs) have grown in prevalence, particular benchmarks have become essential for the evaluation of these models and for understanding model capabilities. M…
Towards Geographic Inclusion in the Evaluation of Text-to-Image Models
Melissa Hall, Samuel J. Bell, Candace Ross +3
Rapid progress in text-to-image generative models coupled with their deployment for visual content creation has magnified the importance of thoroughly evaluating their performance…
[Call for Papers] The 2nd BabyLM Challenge: Sample-efficient pretraining on a developmentally plausible corpus
Leshem Choshen, Ryan Cotterell, Michael Y. Hu +7
After last year's successful BabyLM Challenge, the competition will be hosted again in 2024/2025. The overarching goals of the challenge remain the same; however, some of the compe…
Improving Text-to-Image Consistency via Automatic Prompt Optimization
Oscar Mañas, Pietro Astolfi, Melissa Hall +6
Impressive advances in text-to-image (T2I) generative models have yielded a plethora of high performing models which are able to generate aesthetically appealing, photorealistic im…