Publications (7)
Learning to detect continuous gravitational waves: an open data-analysis competition
Rodrigo Tenorio, Michael J. Williams, Joseph Bayley +29
We report results of a public data-analysis challenge, hosted on the open data-science platform Kaggle, to detect simulated continuous gravitational-wave signals (CWs). These are w…
Position: AI Competitions Provide the Gold Standard for Empirical Rigor in GenAI Evaluation
D. Sculley, Will Cukierski, Phil Culliton +8
In this position paper, we observe that empirical evaluation in Generative AI is at a crisis point since traditional ML evaluation and benchmarking strategies are insufficient to m…
CAFA-evaluator: A Python Tool for Benchmarking Ontological Classification Methods
Damiano Piovesan, Davide Zago, Parnal Joshi +8
We present CAFA-evaluator, a powerful Python program designed to evaluate the performance of prediction methods on targets with hierarchical concept dependencies. It generalizes mu…
Deep learning models for predicting RNA degradation via dual crowdsourcing
Hannah K. Wayment-Steele, Wipapat Kladwang, Andrew M. Watkins +26
Messenger RNA-based medicines hold immense potential, as evidenced by their rapid deployment as COVID-19 vaccines. However, worldwide distribution of mRNA molecules has been limite…
The RSNA-ASNR-MICCAI BraTS 2021 Benchmark on Brain Tumor Segmentation and Radiogenomic Classification
Ujjwal Baid, Satyam Ghodasara, Suyash Mohan +100
The BraTS 2021 challenge celebrates its 10th anniversary and is jointly organized by the Radiological Society of North America (RSNA), the American Society of Neuroradiology (ASNR)…
Challenge design roadmap
Hugo Jair Escalante Balderas, Isabelle Guyon, Addison Howard +2
Challenges can be seen as a type of game that motivates participants to solve serious tasks. As a result, competition organizers must develop effective game rules. However, these r…