From Absolute to Relative Code Comprehensibility Prediction

Abstract

Automatically predicting code comprehensibility could support tasks such as refactoring and code review. Existing metrics correlate poorly with human comprehension, motivating ML models that predict comprehensibility directly from code and developer features. Prior models predict absolute comprehensibility (AC), a comprehensibility value for an isolated snippet, but perform poorly since AC is a subjective proxy for a complex cognitive process. We propose relative comprehensibility (RC) as an alternative task: given two snippets, predict which is easier to understand or whether they are comparable. We hypothesize RC is easier to learn, since it only requires identifying distinguishing features between snippets rather than estimating absolute values. Using 150 Java snippets and 12,540 human comprehensibility measurements from two prior studies, we compare AC and RC prediction across classical ML models, a CNN, and two LLMs, evaluating both snippet-wise (aggregate) and developer-wise (individual-judgment) predictions. AC models rarely beat simple baselines (at most 33.4\% average relative improvement), while snippet-wise RC models outperform baselines in 96.8\% of configurations, with gains up to 159.8\% consistent across architectures, though developer-wise results are more variable. We surveyed 38 practitioners and found both AC and RC useful, with a stronger preference for RC in comparison-oriented tasks like refactoring and review.