Tests of measurement invariance failed to support the application of the "then-test"

Sandra Nolte, Gerald R. Elsworth, Andrew J. Sinclair, Richard H. Osborne

Research output: Contribution to journalArticleResearchpeer-review

38 Citations (Scopus)

Abstract

Objective: The use of then-test (retrospective pre-test) scores has frequently been proposed as a solution to potential confounding of change scores because of response shift, as it is assumed that then-test and post-test responses are provided from the same perspective. However, this assumption has not been formally tested using robust quantitative methods. The aim of this study was to compare the psychometric performance of then-test/post-test with traditional pre-test/post-test data and assessing whether the resulting data structures support the application of the then-test for evaluations of chronic disease self-management interventions. Study Design and Setting: Pre-test, post-test, and then-test data were collected from 314 participants of self-management courses using the Health Education Impact Questionnaire (heiQ). The derived change scores (pre-test/post-test; then-test/post-test) were examined for their psychometric performance using tests of measurement invariance. Results: Few questionnaire items were noninvariant across pre-test/post-test, with four items identified and requiring removal to enable an unbiased comparison of factor means. In contrast, 12 items were identified and required removal in then-test/post-test data to avoid biased change score estimates. Conclusion: Traditional pre-test/post-test data appear to be robust with little indication of response shift. In contrast, the weaker psychometric performance of then-test/post-test data suggests psychometric flaws that may be the result of implicit theory of change, social desirability, and recall bias.

Original languageEnglish
Pages (from-to)1173-1180
Number of pages8
JournalJournal of Clinical Epidemiology
Volume62
Issue number11
DOIs
Publication statusPublished - 1 Nov 2009
Externally publishedYes

Keywords

  • Chronic disease
  • Patient education
  • Program evaluation
  • Response shift
  • Retrospective pre-test
  • Statistical bias

Cite this