Evaluation of distance measures for nonparametric classification in small-scale educational settings

Abstract

Cognitive diagnosis models (CDMs) are special cases of latent class models that characterize the relationship of observable data (typically in the form of questionnaire responses) to a set of categorical latent variables (typically dichotomous). CDMs have been commonly applied in educational testing situations to provide diagnostic information by reporting examinees' mastery profiles on a set of predefined skills or attributes. Parametric CDM estimation, however, requires a fairly large sample size, much larger than what is typical of assessments designed to guide classroom learning. This gap is filled by efficient non-parametric or algorithmic approaches, that classify examinees by minimizing the a distance metric between observed responses and expected responses for a given mastery profile. This paper aims to evaluate a set of distance measures of the $L_2$ distance family for general non-parametric classification in small-scale educational settings. A systematic comparison on simulated data showed that the squared $\chi^2$ distance performed better or equally well compared to the Euclidean squared distance, which is usually the default choice. Moreover, test length is a crucial factor for classification performance, where tests with more items and/or less attributes should be preferred to compensate for small sample sizes. Recommendations for application of non-parametric CDMs in the classroom are provided.