Models in the Same Family are NOT Trust-Equivalent

Abstract

Within a model family, a smaller variant is often deployed as a drop-in replacement for a larger one when their performance is similar. However, performance alone does not tell the full story. We propose a framework to evaluate trust-equivalence between a larger model and a smaller one in the same family along two dimensions. The first is attribution alignment: do both models base their predictions on the same input features? The second is calibration similarity: do both models share the same relationship between confidence and accuracy? We evaluate the Llama-2 family on two text classification tasks: Natural Language Inference and Paraphrase Identification. Attribution alignment is measured using two well-known methods: LIME and SHAP. Agreement between model pairs is quantified via the Jaccard coefficient over top-K attributed features. We observe that attribution alignment between models is generally low, indicating that smaller and larger models base their predictions on different input features. Calibration similarity is assessed using ECE, MCE, Brier Score, and Reliability Diagrams. Calibration profiles differ substantially across model sizes. There is no consistent relationship between model size and calibration quality. We have additionally verified these trends on two encoder-only families: BERT and Vision Transformer. The results are consistent with those reported here. Our experimental results show that replacing a larger model with a smaller one from the same family is a multidimensional decision that requires consideration beyond performance measures alone. Trust-equivalence must be assessed explicitly. It cannot be assumed from performance alone.

Updated title and author list, exapnded experimental results