Do Code Language Models Follow Tests? Paired Interventions on Program Behavior

Yunhao Liang, Chengguang Gan, Ruixuan Ying +3

Visible tests specify concrete program behavior, but an improvement in benchmark accuracy does not establish that a model follows the rule expressed by those tests. We study test u…