Influence Functions for Preference Dataset Pruning

Daniel Fein, Gabriela Aranguiz-Dias

Language models are commonly fine-tuned via reinforcement learning to alter their behavior or elicit new capabilities. Datasets used for these purposes, and particularly human pref…