3 citations · 5 across the 9 of their papers we have counts for
10 papers
GardenBench: A lightweight, daily evaluation of LLM capabilities
Benedict Cummins-Mburu, Mariana Garcia Mejia, Oscar Heath +4
We introduce GardenBench, a benchmark with 11 test suites that evaluate the capabilities of Large Language Models (LLMs) across a range of tasks. By systematically prompting LLMs w…
Correcting Mode Collapse in Silicon Sampling with Semantic Similarity Rating
Oscar Heath, Rohan Alexander
Silicon sampling refers to the use of Large Language Models (LLMs) to generate responses to surveys. It has shown promise, but tends to generate response distributions with unreali…
Same Prompt, Different Outcomes: Evaluating the Reproducibility of Data Analysis by LLMs
Jiaxin Cui, Rohan Alexander
We systematically evaluate the reproducibility of data analysis conducted by Large Language Models (LLMs). We evaluate two prompting strategies, six models, and four temperature se…
Benchmarking AI Performance on End-to-End Data Science Projects
Evelyn Hughes, Rohan Alexander
Data science is an integrated workflow of technical, analytical, communication, and ethical skills, but current AI benchmarks focus mostly on constituent parts. We test whether AI…
Prompting the Professoriate: A Qualitative Study of Instructor Perspectives on LLMs in Data Science Education
Ana Elisa Lopez-Miranda, Tiffany Timbers, Rohan Alexander
Large Language Models (LLMs) have shifted in just a few years from novelty to ubiquity, raising fundamental questions for data science education. Tasks once used to teach coding, w…
Limits to AI Growth: The Ecological and Social Consequences of Scaling
Eshta Bhardwaj, Rohan Alexander, Christoph Becker
The accelerating development and deployment of AI technologies depend on the continued ability to scale their infrastructure. This has implied increasing amounts of monetary invest…
The Increased Effect of Elections and Changing Prime Ministers on Topics Discussed in the Australian Federal Parliament between 1901 and 2018
Rohan Alexander, Monica Alexander
Politics and discussion in parliament is likely to be influenced by the party in power and associated election cycles. However, little is known about the extent to which these even…
Reproducibility of COVID-19 pre-prints
Annie Collins, Rohan Alexander
To examine the reproducibility of COVID-19 research, we create a dataset of pre-prints posted to arXiv, bioRxiv, and medRxiv between 28 January 2020 and 30 June 2021 that are relat…
Detecting Hate Speech with GPT-3
Ke-Li Chiu, Annie Collins, Rohan Alexander
Sophisticated language models such as OpenAI's GPT-3 can generate hateful text that targets marginalized groups. Given this capacity, we are interested in whether large language mo…
On consistency scores in text data with an implementation in R
Ke-Li Chiu, Rohan Alexander
In this paper, we introduce a reproducible cleaning process for the text extracted from PDFs using n-gram models. Our approach compares the originally extracted text with the text…