Bonferonni The and Sidk Corrections for Multiple Comparisons
Herv Abdi1
1 Overview
The more tests we perform on a set of data, the more likely we are to reject the null hypothesis when it is true (i.e., a "Type I" error). This is a consequence of the logic of hypothesis testing: We reject the null hypothesis if we witness a rare event. But the larger the number of tests, the easier it is to find rare events and therefore the easier it is to make the mistake of thinking that there is an effect when there is none. This problem is called the inflation of the alpha level. In order to be protected from it, one strategy is to correct the alpha level when performing multiple tests. Making the alpha level more stringent (i.e., smaller) will create less errors, but it may also make it harder to detect real effects.
2 The different meanings of alpha
Maybe it is because computers make it easier to run statistical analyses that researchers perform more and more statistical tests on a
In: Neil Salkind (Ed.) (2007). Encyclopedia of Measurement and Statistics. Thousand Oaks (CA): Sage. Address correspondence to: Herv Abdi Program in Cognition and Neurosciences, MS: Gr.4.1, The University of Texas at Dallas, Richardson, TX 750830688, USA E-mail: [e-mail] http://www.utd.edu/herve
1
1
Herv Abdi: The Bonferonni and Sidk Corrections
same set of data. For example, brain imaging researchers will routinely run millions of tests to analyze an experiment. Running so many tests increases the risk of false alarms. To illustrate, imagine the following "pseudo-experiment": I toss 20 coins, and I try to force the coins to fall on the heads. I know that, from the "binomial test," the null hypothesis is rejected at the = .05 level if the number of heads is greater than 14. I repeat this experiment 10 times. Suppose that one trial gives the "significant" result of 16 heads versus 4 tails. Did I influence the coins on that occasion? Of course not, because the larger the number of experiments, the greater the probability of detecting a low-probability event (like 16 versus 4). In fact, waiting long enough is a sure way of detecting rare events!
2.1 Probability in the family
A family of tests is the technical term for a series of tests performed on a set of data. In this section we show how to compute the probability of rejecting the null hypothesis at least once in a family of tests when the null hypothesis is true. For convenience, suppose that we set the significance level at =.05. For each test (i.e., one trial in the example of the coins) the probability of making a Type I error is equal to = .05. The events "making a Type I error" and "not making a Type I error" are complementary events (they cannot occur simultaneously). Therefore the probability of not making a Type I error on one trial is equal to 1 - = 1 - .05 = .95 . Recall that when two events are independent, the probability of observing these two events together is the product of their probabilities. Thus, if the tests are independent, the probability of not making a Type I error on the first and the second tests is .95 .95 = (1 - .05)2 = (1 - )2 . 2
Herv Abdi: The Bonferonni and Sidk Corrections
With 3 tests, we find that the probability of not making a Type I error on all tests is: .95 .95 .95 = (1 - .05)3 = (1 - )3 . For a family of C tests, the probability of not making a Type I error for the whole family is: (1 - )C . For our example, the probability of not making a Type I error on the family is (1 - )C = (1 - .05)10 = .599 . Now, what we are looking for is the probability of making one or more Type I errors on the family of tests. This event is the complement of the event not making a Type I error on the family and therefore it is equal to 1 - (1 - )C . For our example, we find 1 - (1 - .05)10 = .401 . So, with an level of .05 for each of the 10 tests, the probability of wrongly rejecting the null hypothesis is .401. This example makes clear the need to distinguish between two meanings of when performing multiple tests: The probability of making a Type I error when dealing only with a specific test. This probability is denoted [P T ] (pronounced "alpha per test"). It is also called the testwise alpha. The probability of making at least one Type I error for the whole family of tests. This probability is denoted [P F ] (pronounced "alpha per family of tests"). It is also called the familywise or the experimentwise alpha.
3
Herv Abdi: The Bonferonni and Sidk Corrections
Table 1: Results of a Monte Carlo simulation. Numbers of Type 1 errors when performing C = 5 tests for 10, 000 families when H0 is true. How to read the table? For example, 192 families over 10, 000 have 2 Type 1 errors, this gives 2 192 = 384 Type 1 errors. Number of families with X Type I errors 7, 868 1, 907 192 20 13 0 10, 000 X : Number of Type I errors per family 0 1 2 3 4 5 Number of Type I errors 0 1, 907 384 60 52 0 2, 403
2.2 A Monte Carlo illustration
A "Monte Carlo" simulation can illustrate the difference between [P T ] and [P F ]. The Monte Carlo technique consists of running a simulated experiment many times using random data. This gives the pattern of results that happens on the basis of chance. Here 6 groups with 100 observations per group were created with data randomly sampled from the same normal population. By construction, H0 is true (i.e., all population means are equal). Call that procedure an experiment. We performed 5 independent tests from these 6 groups. For each test, we computed an F-test. If its probability was smaller than = .05, the test was declared significant (i.e., [P T ] is used). We performed this experiment 10, 000 times. Therefore, there were 10, 000 experiments, 10, 000 families, and 5 10, 000 = 50, 000 tests. The results of this simulation are given in Table 1. Table 1 shows that H0 is rejected for 2, 403 tests over 50, 000 tests performed. From these data, an estimation of [P T ] is computed as: 4
Herv Abdi: The Bonferonni and Sidk Corrections
[P ] T =
number of significant tests total number of tests 2, 403 = .0479 . 50, 000 (1)
=
This value falls close to the theoretical value of = .05. For 7, 868 families, no test reaches significance. Equivalently for 2, 132 families (10, 000-7, 868) at least one Type I error is made. From these data, [P F ] can be estimated as: [P F ] = number of families with at least 1 Type I error total number of families 2, 132 = .2132 . 10, 000 (2)
=
This value falls close to the theoretical value of [P F ] = 1 - (1 - [P T ])C = 1 - (1 - .05)5 = .226 .
2.3 How to correct for multiple tests: Sidk, Bonferonni, Boole, Dunn
Recall that the probability of making as least one Type I error for a family of C tests is [P F ] = 1 - (1 - [P T ])C . This equation can be rewritten as [P T ] = 1 - (1 - [P F ])1/C . This formula--derived assuming independence of the tests--is sometimes called the Sidk equation. It shows that in order to reach a given [P F ] level, we need to adapt the [P T ] values used for each test. Because the Sidk equation involves a fractional power, it is difficult to compute by hand and therefore several authors derived 5
Herv Abdi: The Bonferonni and Sidk Corrections
a simpler approximation which is known as the Bonferonni (the most popular name), or Boole, or even Dunn approximation. Technically, it is the first (linear) term of a Taylor expansion of the Sidk equation. This approximation gives [P T ] [P F ] . C
Sidk and Bonferonni are linked to each other by the inequality [P T ] = 1 - (1 - [P F ])1/C [P F ] . C
They are, in general, very close to each other but the Bonferonni approximation is pessimistic (it always does worse than Sidk equation). Probably because it is easier to compute, the Bonferonni approximation is more well known (and cited more often) than the exact Sidk equation. The Sidk-Bonferonni equations can be used to find the value of [P T ] when [P F ] is fixed. For example, suppose that you want to perform 4 independent tests, and you want to limit the risk of making at least one Type I error to an overall value of [P F ] = .05, you will consider a test significant if its associated probability is smaller than [P T ] = 1 - (1 - [P F ])1/C = 1 - (1 - .05)1/4 = .0127 . With the Bonferonni approximation, a test reaches significance if its associated probability is smaller than [P T ] = [P F ] .05 = = .0125 , C 4
which is very close to the exact value of .0127.
2.4 Correction for non-independent tests
The Sidk equation is derived assuming independence of the tests. When they are not independent, it gives a lower bound (cf. Sidk, 1967; Games, 1977), and then: [P F ] 1 - (1 - [P T ])C . 6
Herv Abdi: The Bonferonni and Sidk Corrections
As previously, we can use a Bonferonni approximation because: [P F ] < C [P T ] . Sidk and Bonferonni are related by the inequality [P F ] 1 - (1 - [P T ])C < C [P T ] . The Sidk and Bonferonni inequalities can also be used to find a correction on [P T ] in order to keep [P F ] fixed. the Sidk inequality gives [P T ] 1 - (1 - [P F ])1/C . This is a conservative approximation, because the following inequality holds: [P T ] 1 - (1 - [P F ])1/C . The Bonferonni approximation gives [P T ] [P F ] . C
2.5 Splitting up [P F ] with unequal slices
With the Bonferonni approximation we can make an unequal allocation of [P F ]. This works because with the Bonferonni approximation, [P F ] is the sum of the individual [P T ]: [P F ] C [P T ] = [P T ] + [P T ] + + [P T ] .
C times
If some tests are judged more important a priori than some others, it is possible to allocate unequally [P F ] (cf. Rosenthal & Rosnow, 1985). For example, suppose we have 3 tests that we want to test with an overall [P F ] = .05, and we think that the first test is the most important of the set. Then we can decide to test it with [P T ] = .04, and share the remaining value .01 = .05 - .04 between the last 2 tests, which will be evaluated each with a value of [P T ] = .005. The overall Type I error for the family is equal to [P F ] = .04 + .005 + .005 = .05 which was indeed the value we set 7
Herv Abdi: The Bonferonni and Sidk Corrections
beforehand. It should be emphasized, however, that the (subjective) importance of the tests and the unequal allocation of the individual [P T ] should be decided a priori for this approach to be statistically valid. An unequal allocation of the [P T ] can also be achieved using the Sidk inequality, but it is more computationally involved.
3 Alternatives to Bonferonni
The Sidk-Bonferonni approach becomes very conservative when the number of comparisons becomes large and when the tests are not independent (e.g., as in brain imaging). Recently, some alternative approaches have been proposed (see Shaffer, 1995, for a review) to make the correction less stringent (e.g., Holm 1979, Hochberg, 1988). A more recent approach redefines the problem by replacing the notion of [P F ] by the false discovery rate (FDR) which is defined as the ratio of the number of Type I errors by the number of significant tests (Benjamini & Hochberg, 1995).
References
[1] Benjamini & Hochberg, (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society, Serie B, 57, 289300. [2] Games, P.A. (1977). An improved t table for simultaneous control on g contrasts. Journal of the American Statistical Association, 72, 531534. [3] Hochberg Y. (1988). A sharper Bonferonni procedure for multiple tests of significance. Biometrika, 75, 800803. [4] Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6, 6570. [5] Rosenthal, R. & Rosnow, R.L. (1985). Contrast analysis: focused comparisons. Boston: Cambridge University Press. [6] Shaffer, J.P. (1995). Multiple Hypothesis Testing Annual Review of Psychology, 46, 561584.
8
Herv Abdi: The Bonferonni and Sidk Corrections
[7] Sidk, Z. (1967). Rectangular confidence region for the means of multivariate normal distributions. Journal of the American Statistical Association, 62, 626633.
9