Multi-method versus single method appraisal of clinical quality indicators for the emergency medical services

BACKGROUND: Quality Indicator (QI) appraisal protocol is a novel methodology that combines multiple appraisal methods to comprehensively assess the 'appropriateness' of QIs for a particular healthcare setting. However, they remain inadequately explored compared to the single appraisal method approach. OBJECTIVES: To describe and test a multi-method QI appraisal protocol versus the single method approach, against a series of QIs previously identified as potentially relevant to the prehospital emergency care setting. METHODS: An appraisal protocol was developed consisting of two categorical-based appraisal methods, combined with the qualitative analysis of the discussion generated during the consensus application of each method. The output of the protocol was assessed and compared with the application and output of each method. Inter-rater reliability (IRR) of each particular method was evaluated prior to group consensus rating. Variation in the number of non-valid QIs and the proportion of non-valid QIs identified between each method and the protocol were compared and assessed. RESULTS: There was mixed IRR of the individual methods. There was similarly low-to-moderate correlation of the results obtained between the particular methods (Spearman's rank correlation = 0.42, P < 0.001). From a series of 104 QIs, 11 non-valid QIs were identified that were shared between the individual methods. A further 19 non-valid QIs were identified and not shared by each method, highlighting the benefits of a multi-method approach. The outcomes were additionally evident in the group discussion analysis, which in and of itself added further input that would not have otherwise been captured by the individual methods alone. CONCLUSION: The utilization of a multi-method appraisal protocol offers multiple benefits, when compared to the single appraisal approach, and can provide the confidence that the outcomes of the appraisal will ensure a strong foundation on which the QI framework can be successfully implemented.