Reward Models Are Secretly Value Functions: Temporally Coherent Reward Modeling

Alex Nikulkov

Reward models in RLHF are trained to score only the final token of a response - a choice that discards rich signal from every intermediate position and produces models whose token-…