Query-efficient model evaluation using cached responses

Hayden Helm, Ben Johnson, Carey Priebe

Evaluating a new model on an existing benchmark is often necessary to understand its behavior before deployment. For modern evaluation frameworks, generating and evaluating a respo…