Cross-Institutional Transfer Learning for Educational Models: Implications for Model Performance, Fairness, and Equity

Abstract

Modern machine learning increasingly supports paradigms that are multi-institutional (using data from multiple institutions during training) or cross-institutional (using models from multiple institutions for inference), but the empirical effects of these paradigms are not well understood. This study investigates cross-institutional learning via an empirical case study in higher education. We propose a framework and metrics for assessing the utility and fairness of student dropout prediction models that are transferred across institutions. We examine the feasibility of cross-institutional transfer under real-world data- and model-sharing constraints, quantifying model biases for intersectional student identities, characterizing potential disparate impact due to these biases, and investigating the impact of various cross-institutional ensembling approaches on fairness and overall model performance. We perform this analysis on data representing over 200,000 enrolled students annually from four universities without sharing training data between institutions. We find that a simple zero-shot cross-institutional transfer procedure can achieve similar performance to locally-trained models for all institutions in our study, without sacrificing model fairness. We also find that stacked ensembling provides no additional benefits to overall performance or fairness compared to either a local model or the zero-shot transfer procedure we tested. We find no evidence of a fairness-accuracy tradeoff across dozens of models and transfer schemes evaluated. Our auditing procedure also highlights the importance of intersectional fairness analysis, revealing performance disparities at the intersection of sensitive identity groups that are concealed under one-dimensional analysis.

Code to reproduce our experiments is available at https://github.com/educational-technology-collective/cross-institutional-transfer-learning-facct-2023

References in corpus (12)

• Deep Learning with Differential Privacy
• Federated Machine Learning: Concept and Applications
• The Measure and Mismeasure of Fairness
• Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization
• Scalable Private Learning with PATE
• Fairness Without Demographics in Repeated Loss Minimization
• Predicting University Students' Academic Success and Major using Random Forests
• Fairness Violations and Mitigation under Covariate Shift
• Auditing and Achieving Intersectional Fairness in Classification Problems
• Fairness with Overlapping Groups
• Subgroup Robustness Grows On Trees: An Empirical Baseline Investigation
• On the limits of algorithmic prediction across the globe

Cited by in corpus (1)

• Gaining Insights into Group-Level Course Difficulty via Differential Course Functioning