Modernizing use of regression models in physics education research: a review of hierarchical linear modeling

Abstract

Physics education researchers (PER) often analyze student data with single-level regression models (e.g., linear and logistic regression). However, education datasets can have hierarchical structures, such as students nested within courses, that single-level models fail to account for. The improper use of single-level models to analyze hierarchical datasets can lead to biased findings. Hierarchical models (a.k.a., multi-level models) account for this hierarchical nested structure in the data. In this publication, we outline the theoretical differences between how single-level and multi-level models handle hierarchical datasets. We then present analysis of a dataset from 112 introductory physics courses using both multiple linear regression and hierarchical linear modeling to illustrate the potential impact of using an inappropriate analytical method on PER findings and implications. Research can leverage multi-institutional datasets to improve the field's understanding of how to support student success in physics. There is no post hoc fix, however, if researchers use inappropriate single-level models to analyze multi-level datasets. To continue developing reliable and generalizable knowledge, PER should use hierarchical models when analyzing hierarchical datasets. The supplemental materials include a sample dataset, R code to model the building and analysis presented in the paper, and an HTML output from the R code.

13 pages, 4 figures, 6 tables, submitted as part of a collection on quantitative methods in physics education research

References in corpus (6)

• Sustaining Educational Reforms in Introductory Physics
• A comparison of Hake's g and Cohen's d for analyzing gains on concept inventories
• Students' epistemologies about experimental physics: Validating the Colorado Learning Attitudes about Science Survey for Experimental Physics
• Extending positive C-LASS results across multiple instructors and multiple classes of Modeling Instruction
• Beyond performance metrics: Examining a decrease in students' physics self-efficacy through a social network lens
• Student Outcomes Across Collaborative Learning Environments

Cited by in corpus (7)

• Associations between learning assistants, passing introductory physics, and equity: A quantitative critical race theory investigation
• Investigating society's educational debts due to racism and sexism in student attitudes about physics using quantitative critical race theory
• Predicting time to graduation at a large enrollment American university
• Bias in physics peer recognition does not explain gaps in perceived recognition
• What topics of peer interactions correlate with student performance in physics courses?
• Sense of agency, gender, and students' perception in open-ended physics labs
• In a physics curriculum only introductory physics course grades show gender differences but do not predict future course performance for physics majors