7: CHAPTER DEALING WITH UNCERTAINTY: EXPECTED VALUE, SENSITIVITY ANALYSIS, AND THE VALUE OF INFORMATION Purpose: Develop the concepts of expected value, sensitivity analysis, and the value of information. EXPECTED VALUE ANALYSIS Expected value analysis consists of modeling uncertainty as contingencies with specific probabilities of occurrence. It begins with the specification of a set of contingencies that are exhaustive and mutually exclusive. In practice, this means the contingencies capture the full range of likely variation in net benefits and accurately represent possible outcomes between the extremes. Once the analyst identifies representative contingencies, the next step is to assign probabilities to each of them. The probabilities must be non-negative and sum to one. The probabilities can be based on historically observed frequencies, subjective assessments, or experts (based on information, theory, or both). Calculating the Expected Value of Net Benefits Calculate the net benefits of each contingency and then multiply by that contingency's probability of occurrence. Then sum all of the weighted benefits. E(NB) = Pi (Bi - Ci) Games Against Nature (Normal Form) have the following elements: states of nature, probabilities of occurrence, actions available to the decision maker facing nature, and payoffs to the decision maker under each combination of state of nature and action. In CBA it is common practice to treat expected values as if they were certain (specific) amounts, even though the actual results rarely equal the expected value. This is not conceptually correct when measuring the WTP in situations where individuals face uncertainty. In practice, however, treating them as commensurate is reasonable when either the pooling of risk over the collection of policies, or the pooling of risk over the collection of persons affected by a policy, will make the actual realized values of costs and benefits close to their expected values. Unpooled risk may require an adjustment to expected net benefits called an option-value, which is addressed in Chapter 8. Decision Trees and Expected Net Benefits Basic expected value analysis takes the weighted average over all contingencies. This can be extended to situations where costs and benefits accrue over several years, as long as the risks in each year are independent of the actions in the previous year. This cannot be done when either the net benefits or probability of a contingency depends on contingencies that have previously occurred. Decision analysis is used in these situations. Decision analysis can be thought of as an extended-form game against nature. It has two stages. First, one specifies the logical structure of the decision problem in terms of sequences of decisions and realizations of contingencies using a diagram (called a decision tree) that links an initial decision to final
Boardman, Greenberg, Vining, Weimer / Cost-Benefit Analysis, 3rd Edition Instructor's Manual 7-1
outcomes. Second, one works backwards from final outcomes to the initial decision, calculating expected values of net benefits across contingencies and pruning dominated branches (ones with lower expected values of net benefits). Vaccine Example: Present value of expected net benefits of the vaccination program is simply E(CNV) - E(CV) (i.e., the expected value of the costs when not implementing the program minus the expected costs when implementing the program). SENSITIVITY ANALYSIS There are several key ideas to sensitivity analysis: We face uncertainty about the predicted impacts and the values assigned to them. Most plausible estimates comprise the base case. The purpose of sensitivity analysis is to show how sensitive predicted net benefits are to changes in assumptions. (If the sign of net benefits doesn't change after considering the range of assumptions, then the analysis is robust and we can have greater confidence in it.) Looking at all combinations of assumptions is infeasible. There are three manageable approaches: 1. Partial sensitivity analysis: Asks, how do net benefits change as one assumption varies (holding other assumptions constant)? It should be used for the most important or uncertain assumptions. 2. Best/worst case analysis: Can be used to find worst and best case scenarios (subset of assumptions). 3. Monte Carlo sensitivity analysis: Creates a distribution of net benefits from drawing key assumptions from a probability distribution, with variance and mean drawn from information on the risk of the project. Partial Sensitivity Analysis The value of a parameter where net benefits switch sign is called the breakeven value. A thorough investigation of sensitivity ideally considers the impact of changes in each of the important assumptions. Best and Worst Case Analysis Base Case: Assign the most plausible numerical values to unknown parameters to produce an estimate of net benefits that is thought to be most representative. Worst Case: Assign the least favorable of the plausible range of values to the parameters. Best Case: Assign the most favorable of the plausible range of values to the parameters.
Worst case analysis is useful as a check against optimistic forecasts and for decision-makers who are risk averse. In worst case scenarios, care must be taken when determining which are the most conservative assumptions. For example, in the vaccine example, under the base case net benefits increase as value of life increases, and in the worst case, net benefits decrease as value of life increases. This change of direction means that what would be the most conservative assumption in the base case would actually be the most favorable assumption in the worst case.
Boardman, Greenberg, Vining, Weimer / Cost-Benefit Analysis, 3rd Edition Instructor's Manual 7-2
Caution is also warranted when net benefits are a non-linear function of a parameter. In this case, the value parameter that maximizes net benefits may not be at the extreme of its range. The relationship of net benefits to a parameter can be determined by inspecting the functional form of the model used to calculate net benefits. Monte Carlo Sensitivity Analysis Partial and best/worst case sensitivity analyses have two limitations. First, they may not take account of all of the available information about the assumed values of parameters (i.e., worst and best cases are highly unlikely). Second, these techniques do not directly provide information about the variance of the statistical distribution of the realized net benefits (i.e., one would feel more confident about an expected value with a smaller variance because it has a higher probability of producing net benefits near the expected value). The essence of Monte Carlo analysis is playing games of chance many times to elicit a distribution of outcomes. It plays an important role in the investigation of statistical estimators whose properties cannot be adequately determined through mathematical techniques alone. Basis steps of Monte Carlo Analysis (MCA): First, specify the probability distributions for all of the important uncertain quantitative assumptions (if no theoretical or empirical evidence suggests a particular distribution, a uniform distribution, if all values are equally likely, or a normal distribution, if a value near the expected value is more plausible, can be used). Second, execute a trial by taking a random draw from the distribution for each parameter to arrive at a specific value for computing realized net benefits. Third, repeat the trial many times. The average of the trials provides an estimate of the expected value of net benefits. An approximation of the probability distribution of net benefits can be obtained by creating a histogram. (As the number of trials approaches infinity, the frequency will converge to the true underlying probability.) Note: If the calculation of net benefits involves sums of random variables, using the expected values of the variables yields expected value of net benefits. If the calculation of net benefits involves sums and products of random variables, using the expected values yields the expected value of net benefits only if the random variables are uncorrelated. In the Monte Carlo approach, correlations can be taken into account by drawing parameter values from either multivariate or conditional distributions rather than from independent univariate distributions. If the calculation involves ratios of random variables, then even independence does not guarantee that their expected values will yield the correct value of net benefits. Trials can be used to directly calculate the sample variance, standard error, and other summary statistics describing net benefits. With MCA, parameters (such as time and life) that are uncertain (but that are treated as certain in the previous example) can be examined. The parameters could be treated as random variables, or the MCA could be repeated for a number of combinations of fixed values of time and life. The result is a collection of histograms that provides a basis for assessing how sensitive our assessment of net benefits is to changes in these critical values. INFORMATION AND QUASI-OPTION VALUE Introduction to the Value of Information
Boardman, Greenberg, Vining, Weimer / Cost-Benefit Analysis, 3rd Edition Instructor's Manual 7-3
The value of information in the context of a game against nature answers the following question: By how much would the information increase the expected value of playing the game? In order to place a value on information, the expected net benefits of the optimal choice in the game without information are compared with the expected net benefits resulting from the optimal choice in the game with information. The value of the information is the difference between the net benefits. The value of information derives from the fact that it leads to different optimal decisions (i.e., if the end decision doesn't change, the value doesn't provide any value). Also, analysts often face choices involving the allocation of resources (such as time, money, and energy) toward reducing uncertainty in the values of the parameters used to calculate net benefits (i.e., use larger sample size). In this case, for the investment of resources to be worthwhile, a meaningful change in the distribution of realized net benefits is necessary. Quasi-Option Value Quasi-option value is the expected value of information gained by delaying an irreversible decision. It can be quantified by formulating a multi-period decision problem that allows for the revelation of information about the value of options in later periods. Exogenous learning: learning is revealed no matter what option is taken. After the first period we discover with a certainty which of the two contingencies will occur. Quasi-option value is the difference in expected net benefits between the learning and no learning case. Endogenous learning: information is generated only through the activity (whatever the program is) itself. This leads Exogenous learning to give large no activity results (i.e., hold off decision) and endogenous learning to give large limited activity results (i.e., limited program). Note: Estimates of the quasi-option values were generated by comparing expected values from the assumed two period decision problem to a one-period decision problem that incorrectly failed to take account of learning. If the correct decision problem is known, however, then there is no need to worry about quasi-option values, as solving the correct decision problem leads to appropriate calculations of expected net benefits. Thus, if you are in a position to calculate quasi-option value, then there is no need to do so.
Boardman, Greenberg, Vining, Weimer / Cost-Benefit Analysis, 3rd Edition Instructor's Manual 7-4