An Evaluation of Item Fit Based on Generalized Residual Item Response Functions

Abstract

Evaluation of item ft for item response theory (IRT) models often involves a comparison of the observed and expected item response functions (IRFs). Several statistics have been suggested for evaluating item ft based on the discrepancy between IRFs, but the asymptotic distributions of the statistics under the null hypothesis are often not well established. Haberman et al. developed a method for evaluating the ft of IRFs based on generalized residuals. These residuals are functions of the latent proficiency variable in the IRT model and follow the standard normal distribution asymptotically. We develop a method to summarize these generalized residuals into a single summary statistic for each item and evaluate its asymptotic distribution. Kondratek suggested a similar Wald-type statistic, but without accounting for the uncertainty in the estimation of the item parameters. Our method combines the work of Haberman and Kondratek, resulting in a single ft statistic per item while accounting for estimation error. A series of simulations was carried out to investigate the performance of our statistic and compare it to several popular item ft statistics. Our method resulted in similar Type I errors as Kondratek’s statistic, with slightly better results in the case of small samples. Furthermore, the recovery was consistent across different levels of item difficulty, and power of the new item ft statistic was relatively low, except for problematic individual items, but this result was found with two competing statistics as well.

Suggested citation: Liao, X., van Rijn, P., & Sinharay, S. (2025). An evaluation of item fit based on generalized residual item response functions (Research Report No. RR-25-13). ETS. https://doi.org/10.64634/b68vz316