Development and validation of SEER (Seeking, Engaging with and Evaluating Research)

A measure of policymakers' capacity to engage with and use research

Sue E. Brennan, Joanne E. McKenzie, Tari Turner, Sally Redman, Steve Makkar, Anna Williamson, Abby Haynes, Sally E. Green

Research output: Contribution to journalArticleResearchpeer-review

9 Citations (Scopus)

Abstract

Background: Capacity building strategies are widely used to increase the use of research in policy development. However, a lack of well-validated measures for policy contexts has hampered efforts to identify priorities for capacity building and to evaluate the impact of strategies. We aimed to address this gap by developing SEER (Seeking, Engaging with and Evaluating Research), a self-report measure of individual policymakers' capacity to engage with and use research. Methods: We used the SPIRIT Action Framework to identify pertinent domains and guide development of items for measuring each domain. Scales covered (1) individual capacity to use research (confidence in using research, value placed on research, individual perceptions of the value their organisation places on research, supporting tools and systems), (2) actions taken to engage with research and researchers, and (3) use of research to inform policy (extent and type of research use). A sample of policymakers engaged in health policy development provided data to examine scale reliability (internal consistency, test-retest) and validity (relation to measures of similar concepts, relation to a measure of intention to use research, internal structure of the individual capacity scales). Results: Response rates were 55% (150/272 people, 12 agencies) for the validity and internal consistency analyses, and 54% (57/105 people, 9 agencies) for test-retest reliability. The individual capacity scales demonstrated adequate internal consistency reliability (alpha coefficients > 0.7, all four scales) and test-retest reliability (intra-class correlation coefficients > 0.7 for three scales and 0.59 for fourth scale). Scores on individual capacity scales converged as predicted with measures of similar concepts (moderate correlations of > 0.4), and confirmatory factor analysis provided evidence that the scales measured related but distinct concepts. Items in each of these four scales related as predicted to concepts in the measurement model derived from the SPIRIT Action Framework. Evidence about the reliability and validity of the research engagement actions and research use scales was equivocal. Conclusions: Initial testing of SEER suggests that the four individual capacity scales may be used in policy settings to examine current capacity and identify areas for capacity building. The relation between capacity, research engagement actions and research use requires further investigation.

Original languageEnglish
Article number1
Number of pages19
JournalHealth Research Policy and Systems
Volume15
Issue number1
DOIs
Publication statusPublished - 17 Jan 2017

Keywords

  • Capacity building
  • Capacity to use research
  • Conceptual framework
  • Evidence-informed policy
  • Health policy
  • Knowledge exchange
  • Knowledge translation
  • Measurement instrument
  • Questionnaire
  • Research utilisation

Cite this

@article{90ec551a44ff4c4b973c92246e728184,
title = "Development and validation of SEER (Seeking, Engaging with and Evaluating Research): A measure of policymakers' capacity to engage with and use research",
abstract = "Background: Capacity building strategies are widely used to increase the use of research in policy development. However, a lack of well-validated measures for policy contexts has hampered efforts to identify priorities for capacity building and to evaluate the impact of strategies. We aimed to address this gap by developing SEER (Seeking, Engaging with and Evaluating Research), a self-report measure of individual policymakers' capacity to engage with and use research. Methods: We used the SPIRIT Action Framework to identify pertinent domains and guide development of items for measuring each domain. Scales covered (1) individual capacity to use research (confidence in using research, value placed on research, individual perceptions of the value their organisation places on research, supporting tools and systems), (2) actions taken to engage with research and researchers, and (3) use of research to inform policy (extent and type of research use). A sample of policymakers engaged in health policy development provided data to examine scale reliability (internal consistency, test-retest) and validity (relation to measures of similar concepts, relation to a measure of intention to use research, internal structure of the individual capacity scales). Results: Response rates were 55{\%} (150/272 people, 12 agencies) for the validity and internal consistency analyses, and 54{\%} (57/105 people, 9 agencies) for test-retest reliability. The individual capacity scales demonstrated adequate internal consistency reliability (alpha coefficients > 0.7, all four scales) and test-retest reliability (intra-class correlation coefficients > 0.7 for three scales and 0.59 for fourth scale). Scores on individual capacity scales converged as predicted with measures of similar concepts (moderate correlations of > 0.4), and confirmatory factor analysis provided evidence that the scales measured related but distinct concepts. Items in each of these four scales related as predicted to concepts in the measurement model derived from the SPIRIT Action Framework. Evidence about the reliability and validity of the research engagement actions and research use scales was equivocal. Conclusions: Initial testing of SEER suggests that the four individual capacity scales may be used in policy settings to examine current capacity and identify areas for capacity building. The relation between capacity, research engagement actions and research use requires further investigation.",
keywords = "Capacity building, Capacity to use research, Conceptual framework, Evidence-informed policy, Health policy, Knowledge exchange, Knowledge translation, Measurement instrument, Questionnaire, Research utilisation",
author = "Brennan, {Sue E.} and McKenzie, {Joanne E.} and Tari Turner and Sally Redman and Steve Makkar and Anna Williamson and Abby Haynes and Green, {Sally E.}",
year = "2017",
month = "1",
day = "17",
doi = "10.1186/s12961-016-0162-8",
language = "English",
volume = "15",
journal = "Health Research Policy and Systems",
issn = "1478-4505",
publisher = "Springer-Verlag London Ltd.",
number = "1",

}

Development and validation of SEER (Seeking, Engaging with and Evaluating Research) : A measure of policymakers' capacity to engage with and use research. / Brennan, Sue E.; McKenzie, Joanne E.; Turner, Tari; Redman, Sally; Makkar, Steve; Williamson, Anna; Haynes, Abby; Green, Sally E.

In: Health Research Policy and Systems, Vol. 15, No. 1, 1, 17.01.2017.

Research output: Contribution to journalArticleResearchpeer-review

TY - JOUR

T1 - Development and validation of SEER (Seeking, Engaging with and Evaluating Research)

T2 - A measure of policymakers' capacity to engage with and use research

AU - Brennan, Sue E.

AU - McKenzie, Joanne E.

AU - Turner, Tari

AU - Redman, Sally

AU - Makkar, Steve

AU - Williamson, Anna

AU - Haynes, Abby

AU - Green, Sally E.

PY - 2017/1/17

Y1 - 2017/1/17

N2 - Background: Capacity building strategies are widely used to increase the use of research in policy development. However, a lack of well-validated measures for policy contexts has hampered efforts to identify priorities for capacity building and to evaluate the impact of strategies. We aimed to address this gap by developing SEER (Seeking, Engaging with and Evaluating Research), a self-report measure of individual policymakers' capacity to engage with and use research. Methods: We used the SPIRIT Action Framework to identify pertinent domains and guide development of items for measuring each domain. Scales covered (1) individual capacity to use research (confidence in using research, value placed on research, individual perceptions of the value their organisation places on research, supporting tools and systems), (2) actions taken to engage with research and researchers, and (3) use of research to inform policy (extent and type of research use). A sample of policymakers engaged in health policy development provided data to examine scale reliability (internal consistency, test-retest) and validity (relation to measures of similar concepts, relation to a measure of intention to use research, internal structure of the individual capacity scales). Results: Response rates were 55% (150/272 people, 12 agencies) for the validity and internal consistency analyses, and 54% (57/105 people, 9 agencies) for test-retest reliability. The individual capacity scales demonstrated adequate internal consistency reliability (alpha coefficients > 0.7, all four scales) and test-retest reliability (intra-class correlation coefficients > 0.7 for three scales and 0.59 for fourth scale). Scores on individual capacity scales converged as predicted with measures of similar concepts (moderate correlations of > 0.4), and confirmatory factor analysis provided evidence that the scales measured related but distinct concepts. Items in each of these four scales related as predicted to concepts in the measurement model derived from the SPIRIT Action Framework. Evidence about the reliability and validity of the research engagement actions and research use scales was equivocal. Conclusions: Initial testing of SEER suggests that the four individual capacity scales may be used in policy settings to examine current capacity and identify areas for capacity building. The relation between capacity, research engagement actions and research use requires further investigation.

AB - Background: Capacity building strategies are widely used to increase the use of research in policy development. However, a lack of well-validated measures for policy contexts has hampered efforts to identify priorities for capacity building and to evaluate the impact of strategies. We aimed to address this gap by developing SEER (Seeking, Engaging with and Evaluating Research), a self-report measure of individual policymakers' capacity to engage with and use research. Methods: We used the SPIRIT Action Framework to identify pertinent domains and guide development of items for measuring each domain. Scales covered (1) individual capacity to use research (confidence in using research, value placed on research, individual perceptions of the value their organisation places on research, supporting tools and systems), (2) actions taken to engage with research and researchers, and (3) use of research to inform policy (extent and type of research use). A sample of policymakers engaged in health policy development provided data to examine scale reliability (internal consistency, test-retest) and validity (relation to measures of similar concepts, relation to a measure of intention to use research, internal structure of the individual capacity scales). Results: Response rates were 55% (150/272 people, 12 agencies) for the validity and internal consistency analyses, and 54% (57/105 people, 9 agencies) for test-retest reliability. The individual capacity scales demonstrated adequate internal consistency reliability (alpha coefficients > 0.7, all four scales) and test-retest reliability (intra-class correlation coefficients > 0.7 for three scales and 0.59 for fourth scale). Scores on individual capacity scales converged as predicted with measures of similar concepts (moderate correlations of > 0.4), and confirmatory factor analysis provided evidence that the scales measured related but distinct concepts. Items in each of these four scales related as predicted to concepts in the measurement model derived from the SPIRIT Action Framework. Evidence about the reliability and validity of the research engagement actions and research use scales was equivocal. Conclusions: Initial testing of SEER suggests that the four individual capacity scales may be used in policy settings to examine current capacity and identify areas for capacity building. The relation between capacity, research engagement actions and research use requires further investigation.

KW - Capacity building

KW - Capacity to use research

KW - Conceptual framework

KW - Evidence-informed policy

KW - Health policy

KW - Knowledge exchange

KW - Knowledge translation

KW - Measurement instrument

KW - Questionnaire

KW - Research utilisation

UR - http://www.scopus.com/inward/record.url?scp=85011407813&partnerID=8YFLogxK

U2 - 10.1186/s12961-016-0162-8

DO - 10.1186/s12961-016-0162-8

M3 - Article

VL - 15

JO - Health Research Policy and Systems

JF - Health Research Policy and Systems

SN - 1478-4505

IS - 1

M1 - 1

ER -