800 Bioinformatics Module 2 Lecture 5
March 17, 2009
Kerby Shedden Department of Statistics [e-mail]
1
Review of simple hypothesis testing Identify a null-hypothesis (that nothing interesting is going on) and (optionally) a set of alternative hypotheses of interest. Construct a test statistic T from the data. T should have these properties: T is close to zero when the data support the null hypothesis; T gets further from zero when the data support the alternative hypothesis. Under the null hypothesis, if we repeatedly sample independent data sets and recalculate T for each set, the variation in T should follow a known distribution (usually standard normal). Suppose Tobs is the observed value of T . The p-value is the probability that |T | is greater than |Tobs | if the null hypothesis is true. Small p-values imply more evidence in the data against the null hypothesis, larger p-values imply more evidence in the data in support of the null hypothesis.
2
Errors in hypothesis testing A "false positive" occurs when the null hypothesis is true, but is rejected. This is also called a "type-I" error. A "false negative" occurs when the alternative hypothesis is true, but the null hypothesis is not rejected. This is also called a "type-II" error. Classical hypothesis testing is set up so that false positives occur a fixed fraction of the time (usually 5%). The probabilty of a false negative occuring depends on the effect size, and is equal to 1 minus the power.
3
Multiple testing Suppose we test a family of hypotheses, e.g. we collect 10 possible biomarkers for lung cancer recurrence, and we perform these tests: Marker 1 does (alternative) or does not (null) associate with recurrence. Marker 2 does (alternative) or does not (null) associate with recurrence. Marker 3 does (alternative) or does not (null) associate with recurrence. If perform all 10 of these tests, and if the null hypotheses are all true, and if the test statistics are independent, the probability that we make at least one false positive statement is around 0.4. This is called the "family-wise error rate." false-positive rate (which is usually 5%). It is always greater than the
4
Examples of multiple testing Genome-wide association studies (GWAS) test up to several million genetic variants for association with a trait. Molecular library screening test a large set of compounds for activity in a high throughput assay (for enzyme inhibition, toxicity, binding to a receptor, ...). Differential gene expression analysis test a large number of genes for differential expression between two classes of samples (e.g. disease vs. healthy). Combination therapy evaluation test various dose combinations of several drugs for safety or efficacy. Biomarker evaluation test a panel of biomarkers for diagnostic or prognostic performance.
5
Multiple testing When many tests are performed, we can ask the following questions: Are there any true positives? How many true positives are there? Which are the true positives?
How you approach things should depend on your goals: surveillance, discovery, validation, estimation, . . .
6
Scales of multiple testing "Classical multiple testing": typically, < 10 tests are performed looking at different treatment contrasts, dose combinations, etc. "Large scale multiple testing": testing all genes, all proteins, etc.
7
Answering the question "are there any true positives?" For large scale testing, the simplest approach is to assess the uniformity of the p-values. If all the null hypotheses are true, test the statistics should be uniformly distributed in the interval (0,1).
8
One way of seeing that the p-values are uniformly distributed when all tested items are truly negative is to plot the "empirical CDF" of the p-values:
1.2 1.0 0.8 0.6 0.4 0.2 0.0 -0.20
m=100
20
Expected value Expected value 2SD
40
Rank
60
80
100
1.2 1.0 0.8 0.6 0.4 0.2 0.0 -0.20
m=1000
p
p
200
Expected value Expected value 2SD
400
Rank
600
800
1000
9
Quantile analysis of the test statistics 1000 independent tests, 20 non-nulls (effect size = ) on left, zero non-nulls on right. Ten replications plotted in each graph.
6
Quantiles of test statistic
2 0 -2
-4 -6-6
Standard normal quantiles
-4
-2
0
2
4
Quantiles of test statistic
4
6
4 3 2 1 0 -1 -2 -3 -4-4
-3
Standard normal quantiles
10
-2 -1
0
1
2
3
4
Controlling family-wise error: Bonferroni approach Without assuming independence of the tests, we get the inequality
P0 (T1 > T or T2 > T or Tm > T ) m. So if we carry out each test at level
= /m then the family-wise error rate is at most . If = 0.05 and m = 100 then = 0.0005. Simple recipe: multiply your p-values by the number of tests (and round down to 1 if the result is greater than 1).
11
False Discovery Rate Suppose we have test statistics Z1 , Z2 , . . . , Zm (i.e. one test per gene in a GWAS, or one test per compound in a screen for enyzme inhibitors). We can sort the test statistics and specify a "tail region" that defines our objects (genes, compounds) of interest. Within the tail region, there is a certain number of "true positive" and "false positive" calls. These are denoted as TP:FP below.
-8 -8 -8
-6 -6 -6
-4 -4 -4
-2 -2 -2
0 0 0
10000:45500, FDR=0.82 6 8 2 4 5000:2700, FDR=0.28 6 8 2 4 1000:63, FDR=0.06 6 8 2 4
The false discovery rate is F P/(T P + F P ) where F P is the number of false positive calls and T P is the number of true positive calls.
12
Estimating the False Discovery Rate The FDR is defined in terms of FP and TP, but we only know FP+TP from our data. To estimate the FDR we have to use the statistical properties of our test (i.e. that the test ...