Reliability-Targeted Simulation of Item Response Data: Solving the Inverse Design Problem

Abstract

Monte Carlo studies of item response theory (IRT) methods often leave the information level implicit in the choice of item parameters. Designs with the same sample size and test length can therefore represent quite different measurement conditions. We develop a calibration procedure that makes the information level an explicit design choice. The target is an information-based reliability index derived from population mean test information. Before generating responses, we solve for a common multiplier on item discriminations, keeping difficulties, test length and the reference population fixed. The R package IRTsimrel implements the procedure, allowing researchers to compare methods across chosen information levels or compare designs at a common target. In an item-level treatment-effect simulation, power increases across targets, while excess rejection from ignoring item-effect variation persists. These comparisons show how targeting helps distinguish findings that depend on the chosen information level from those that persist across the studied range.