4 citations · 14 across the 13 of their papers we have counts for
13 papers
Aggregation Hides Out-of-Distribution Generalization Failures from Spurious Correlations
Olawale Salaudeen, Haoran Zhang, Kumail Alhamoud +2
Benchmarks for out-of-distribution (OOD) generalization frequently show a strong positive correlation between in-distribution (ID) and OOD accuracy across models, termed "accuracy-…
A Definition of AGI
Dan Hendrycks, Dawn Song, Christian Szegedy +30
The lack of a concrete definition for Artificial General Intelligence (AGI) obscures the gap between today's specialized AI and human-level cognition. This paper introduces a quant…
Position: Stop Evaluating AI with Human Tests, Develop Principled, AI-specific Tests instead
Tom Sühr, Florian E. Dorner, Olawale Salaudeen +2
Large Language Models (LLMs) have achieved remarkable results on a range of standardized tests originally designed to assess human cognitive and psychological traits, such as intel…
Measurement to Meaning: A Validity-Centered Framework for AI Evaluation
Olawale Salaudeen, Anka Reuel, Ahmed Ahmed +6
While the capabilities and utility of AI systems have advanced, rigorous norms for evaluating these systems have lagged. Grand claims, such as models achieving general reasoning ca…
Understanding challenges to the interpretation of disaggregated evaluations of algorithmic fairness
Stephen R. Pfohl, Natalie Harris, Chirag Nagpal +12
Disaggregated evaluation across subgroups is critical for assessing the fairness of machine learning models, but its uncritical use can mislead practitioners. We show that equal pe…
Are Domain Generalization Benchmarks with Accuracy on the Line Misspecified?
Olawale Salaudeen, Nicole Chiou, Shiny Weng +1
Spurious correlations, unstable statistical shortcuts a model can exploit, are expected to degrade performance out-of-distribution (OOD). However, across many popular OOD generaliz…
Toward an Evaluation Science for Generative AI Systems
Laura Weidinger, Inioluwa Deborah Raji, Hanna Wallach +7
There is an increasing imperative to anticipate and understand the performance and safety of generative AI systems in real-world deployment contexts. However, the current evaluatio…
What's in a Query: Polarity-Aware Distribution-Based Fair Ranking
Aparna Balagopalan, Kai Wang, Olawale Salaudeen +2
Machine learning-driven rankings, where individuals (or items) are ranked in response to a query, mediate search exposure or attention in a variety of safety-critical settings. Thu…
Causally Inspired Regularization Enables Domain General Representations
Olawale Salaudeen, Sanmi Koyejo
Given a causal graph representing the data-generating process shared across different domains/distributions, enforcing sufficient graph-implied conditional independencies can ident…
ImageNot: A contrast with ImageNet preserves model rankings
Olawale Salaudeen, Moritz Hardt
We introduce ImageNot, a dataset constructed explicitly to be drastically different than ImageNet while matching its scale. ImageNot is designed to test the external validity of de…
Proxy Methods for Domain Adaptation
Katherine Tsai, Stephen R. Pfohl, Olawale Salaudeen +5
We study the problem of domain adaptation under distribution shift, where the shift is due to a change in the distribution of an unobserved, latent variable that confounds both the…
Target Conditioned Representation Independence (TCRI); From Domain-Invariant to Domain-General Representations
Olawale Salaudeen, Oluwasanmi Koyejo
We propose a Target Conditioned Representation Independence (TCRI) objective for domain generalization. TCRI addresses the limitations of existing domain generalization methods due…
Adapting to Latent Subgroup Shifts via Concepts and Proxies
Ibrahim Alabdulmohsin, Nicole Chiou, Alexander D'Amour +7
We address the problem of unsupervised domain adaptation when the source domain differs from the target domain because of a shift in the distribution of a latent subgroup. When thi…