YFPO: Yoked Feature Preference Optimization with Neuron-Guided Rewards

Abstract

Preference optimization has become a widely used post-training paradigm for improving the reasoning abilities of large language models. Existing methods typically learn from preferred and dispreferred responses as external behavioral supervision, while largely ignoring capability-related signals encoded in the model's internal representations. In this work, we study whether such internal signals can provide useful auxiliary supervision for mathematical reasoning. We introduce YFPO (Yoked Feature Preference Optimization), a neuron-guided preference optimization framework that couples response-level preference learning with neuron-level rewards. YFPO first uses AttnLRP to identify math-associated internal features, and then derives an auxiliary reward from the activation margin of these neurons between preferred and dispreferred responses. This reward is combined with the standard preference optimization objective, encouraging the model to align external preferences with internal math-related features. We conduct small-scale experiments on GSM8K with a compact language model. Results show that neuron-guided rewards can influence preference optimization dynamics and yield measurable improvements in several settings, suggesting that internal representations can serve as lightweight and interpretable signals for reasoning-oriented post-training.

Accepted to Findings of AACL-IJCNLP 2026. Camera-ready revision