Empirical evidence of design-related bias in studies of diagnostic tests

Jeroen G. Lijmer, Ben Willem Mol, Siem Heisterkamp, Gouke J. Bonsel, Martin H. Prins, Jan H.P. Van Der Meulen, Patrick M.M. Bossuyt

Research output: Contribution to journalArticleResearchpeer-review

1347 Citations (Scopus)

Abstract

Context: The literature contains a large number of potential biases in the evaluation of diagnostic tests. Strict application of appropriate methodological criteria would invalidate the clinical application of most study results. Objective: To empirically determine the quantitative effect of study design shortcomings on estimates of diagnostic accuracy. Design and Setting: Observational study of the methodological features of 184 original studies evaluating 218 diagnostic tests. Meta-analyses on diagnostic tests were identified through a systematic search of the literature using MEDLINE, EMBASE, and DARE databases and the Cochrane Library (1996-1997). Associations between study characteristics and estimates of diagnostic accuracy were evaluated with a regression model. Main Outcome Measures: Relative diagnostic odds ratio (RDOR), which compared the diagnostic odds ratios of studies of a given test that lacked a particular methodological feature with those without the corresponding shortcomings in design. Results: Fifteen (6.8%) of 218 evaluations met all 8 criteria; 64 (30%) met 6 or more. Studies evaluating tests in a diseased population and a separate control group overestimated the diagnostic performance compared with studies that used a clinical population (RDOR, 3.0; 95% confidence interval [CI], 2.0-4.5). Studies in which different reference tests were used for positive and negative results of the test under study overestimated the diagnostic performance compared with studies using a single reference test for all patients (RDOR, 2.2; 95% CI, 1.5-3.3). Diagnostic performance was also overestimated when the reference test was interpreted with knowledge of the test result (RDOR, 1.3; 95% CI, 1.0-1.9), when no criteria for the test were described (RDOR, 1.7; 95% CI, 1.1-2.5), and when no description of the population under study was provided (RDOR, 1.4; 95% CI, 1.1-1.7). Conclusion: These data provide empirical evidence that diagnostic studies with methodological shortcomings may overestimate the accuracy of a diagnostic test, particularly those including nonrepresentative patients or applying different reference standards.

Original languageEnglish
Pages (from-to)1061-1066
Number of pages6
JournalJAMA
Volume282
Issue number11
DOIs
Publication statusPublished - 15 Sep 1999
Externally publishedYes

Cite this

Lijmer, J. G., Mol, B. W., Heisterkamp, S., Bonsel, G. J., Prins, M. H., Van Der Meulen, J. H. P., & Bossuyt, P. M. M. (1999). Empirical evidence of design-related bias in studies of diagnostic tests. JAMA, 282(11), 1061-1066. https://doi.org/10.1001/jama.282.11.1061