From Value Conditioning to Behavioral Shift: Lightweight Value Alignment of LLMs

Abstract

Large language models exhibit value-related preferences in their outputs, yet steering such behavior often requires large training datasets or paired preference annotations. In this work, we investigate a lightweight alternative: can fine-tuning on value survey responses induce measurable shifts in value-conditioned model outputs and downstream behavior? We first estimate baseline value profiles for several open-source LLMs by asking them to rate value-related descriptions organized around the ten Schwartz values. We then fine-tune models on survey-style supervision that selectively changes the target value ratings while keeping non-target values close to their baseline responses. We evaluate the resulting shifts in two settings. First, we measure whether the intervention changes responses on held-out survey questions. Second, we test whether these survey-induced shifts transfer to out-of-domain behavioral proxies, including a contextual moral-judgment benchmark constructed from Reddit AITA posts and a text-based interactive decision-making benchmark. Our results show that survey-based supervision can change held-out survey responses and produce downstream shifts in model response distributions on proxy behavioral evaluations. However, this transfer is strongly model- and value-dependent, and is shaped by how value descriptions are framed in the supervision data. These findings suggest that value surveys can serve not only as diagnostic tools, but also as lightweight supervision for studying behavioral value conditioning in LLMs.

arXiv comment10 pages 2 figures