Investigate Discuss Estimate Aggregate for structured expert judgement

A. M. Hanea, M. F. McBride, M. A. Burgman, B. C. Wintle, F. Fidler, L. Flander, C. R. Twardy, B. Manning, S. Mascaro

Research output: Contribution to journalArticleResearchpeer-review

47 Citations (Scopus)


This study presents the results of an approach to the prediction of the outcomes of geopolitical events, which we term the IDEA protocol. The participants investigate the background and causal factors behind a question, predict the outcome, and discuss their thinking with others. They then make a second, private and anonymous judgement of the probability of the event, which is subsequently aggregated mathematically. The method performed well relative to both an equally weighted linear pool and a prediction market, and is relatively simple to implement. The results indicate the value of discussion for removing arbitrary linguistic uncertainty and for sharing and debating knowledge, thereby improving the judgements. Weighting individual judgements based on prior performance using Cooke's method improved group judgements. Even though some of the results are not statistically significant, the study may not have had sufficient power to detect some important effects. Nevertheless, the results help us to formulate conjectures, which can then be investigated further.

Original languageEnglish
Pages (from-to)267-279
Number of pages13
JournalInternational Journal of Forecasting
Issue number1
Publication statusPublished - Jan 2017
Externally publishedYes


  • Aggregation
  • Elicitation
  • Performance
  • Probability forecasts
  • Structured protocol

Cite this