Oppenheimer et al 2009 the probable net effect is

(Oppenheimer et al., 2009). The probable net effect is that we were much more liberal in our
inclusion criteria in Experiments 1a and 1b than we were in Experiment 2, and yet we obtained
highly consistent results.
Furthermore, recall that confidence interval construction is hypothesized to benefit the accuracy
of best estimates by improving the recruitment of relevant evidence pertinent to testing the
equivalent of best- and worst-case scenarios or multiple viewpoints (Hemming et al., 2018).
Presumably, the benefit afforded to best-estimate accuracy depends on how accurately the
preceding intervals are constructed. Following this line of reasoning, one might also expect the
correlation between best-estimate accuracy and (calibrated) confidence interval accuracy to be
stronger if interval construction preceded best-estimate construction than if the best estimates
were constructed first. However, we do not find support for that prediction either. Across
experiments and question types, the correlation between GMAE for the best estimates and the
proportion of correct responses in the calibrated confidence interval was r (289) = − .40 ( p < .001)
when best estimates were elicited first and r (257) = − .34 ( p < .001) when they were elicited after
the confidence intervals were constructed. The difference is not significant, z = − 0.79, p = .21.
From a practical perspective, the present results do not indicate the utility of prior confidence
interval construction for improving the accuracy of best estimates. As we noted in the
Introduction, improving the accuracy of judgments represents a major effort that has important
implications for several domains. An important consideration in these efforts is cost. IBBE
protocols require significant additional time (e.g.., in the case of the four-step method, three
additional judgments). Thus, even a small benefit may not justify the added effort, given that
other methods that require a similar number of elicitations have yielded large improvements in
probability judgment accuracy. Notwithstanding the risks associated with internal meta-analyses
(Ueno, Fastrich & Murayama, 2016; Vosgerau, Simonsohn, Nelson & Simmons, 2019), it is
useful to estimate the overall effect size of the elicitation order manipulation we conducted
across three experiments. There is a small positive effect (Cohen’s d = 0.285, 95% CI [0.117,
0.453]) of the modified four-step method we tested on the accuracy of best estimates.
1
Moreover,
in some elicitation contexts, such as decision analysis (Clemen, 1996; von Winterfeldt &
Edwards, 1986), it may be highly desirable, if not necessary, to collect lower- and upper-bound
estimates, in which case there may be a small benefit to following the ordering prescribed by
IBBE protocols. However, if the aim of the method is to improve best estimates, then query
1
Estimation of the confidence interval on d was computed using the implementation of
procedures by Smithson (2001) provided by Wuensch (2012).