6 papers
Let's Measure Information Step-by-Step: AI-Based Evaluation Beyond Vibes
Zachary Robertson, Sanmi Koyejo
We evaluate artificial intelligence (AI) systems without ground truth by exploiting a link between strategic gaming and information loss. Building on established information theory…
A Global Characterization of -Divergences Yielding PSD Mutual-Information Matrices
Zachary Robertson
Given random variables, when does the matrix of pairwise -mutual informations define a PSD kernel over variables? For convex finite generators wi…
Value Alignment of Social Media Ranking Algorithms
Farnaz Jahanbakhsh, Dora Zhao, Tiziano Piccardi +4
While social media feed rankings are primarily driven by engagement signals rather than any explicit value system, the resulting algorithmic feeds are not value-neutral: engagement…
Identity-Link IRT for Label-Free LLM Evaluation: Preserving Additivity in TVD-MI Scores
Zachary Robertson
Pairwise comparisons of large language models using total variation distance mutual information (TVD-MI) produce binary critic decisions per pair. We show that averaging TVD-MI's b…
Measurement to Meaning: A Validity-Centered Framework for AI Evaluation
Olawale Salaudeen, Anka Reuel, Ahmed Ahmed +6
While the capabilities and utility of AI systems have advanced, rigorous norms for evaluating these systems have lagged. Grand claims, such as models achieving general reasoning ca…
Implicit Regularization in Feedback Alignment Learning Mechanisms for Neural Networks
Zachary Robertson, Oluwasanmi Koyejo
Feedback Alignment (FA) methods are biologically inspired local learning rules for training neural networks with reduced communication between layers. While FA has potential applic…