7 citations · 14 across the 7 of their papers we have counts for
7 papers
Soft Contamination Means Benchmarks Test Shallow Generalization
Ari Spiesberger, Juan J. Vazquez, Nicky Pochinkov +4
If LLM training data is polluted with benchmark test data, then benchmark performance gives biased estimates of out-of-distribution (OOD) generalization. Typical decontamination fi…
How much technical talent is there? A systematic estimate of the ML research pool among 3 million consultants
Maximilian Schons, Red Bermejo, Florian Aldehoff-Zeidler +4
We identify a substantial pool of technically competent ML research talent (in the low thousands) in companies which offer consulting in machine learning. We systematically searche…
Questionable practices in machine learning
Gavin Leech, Juan J. Vazquez, Niclas Kupper +2
Evaluating modern ML models is hard. The strong incentive for researchers and companies to report a state-of-the-art result on some metric often leads to questionable research prac…
Ten Hard Problems in Artificial Intelligence We Must Get Right
Gavin Leech, Simson Garfinkel, Misha Yagudin +2
We explore the AI2050 "hard problems" that block the promise of AI and cause AI risks: (1) developing general capabilities of the systems; (2) assuring the performance of AI system…
Steering Language Models With Activation Engineering
Alexander Matt Turner, Lisa Thiergart, Gavin Leech +4
Prompt engineering and finetuning aim to maximize language model performance on a given metric (like toxicity reduction). However, these methods do not fully elicit a model's capab…
Massively Parallel Reweighted Wake-Sleep
Thomas Heap, Gavin Leech, Laurence Aitchison
Reweighted wake-sleep (RWS) is a machine learning method for performing Bayesian inference in a very general class of models. RWS draws samples from an underlying approximate p…
Decision trees compensate for model misspecification
Hugh Panton, Gavin Leech, Laurence Aitchison
The best-performing models in ML are not interpretable. If we can explain why they outperform, we may be able to replicate these mechanisms and obtain both interpretability and per…