500 Stat - Notes and example of false discovery rate adjustment
These notes discuss the Benjamini and Hochberg 1995 false discovery rate adjustment. It is very widely used today, especially in high throughput genomics. An extension, Storey's positive false discovery rate, pFDR or q-value, (described in Storey and Tibshirani 2003) is less conservative and increasing in popularity. Context and definition of FDR: You are comparing mean expression levels for a gene under two treatments. You are interested in the genes that are differentially expressed, i.e. 1 = 2 . You have a decision procedure (e.g. an unadjusted t-test) that declares a gene differentially expressed (or significant). The decision procedure could be to do a t-test and declare a gene 'significant' if p < 0.05. This procedure results in a list of genes considered significant, each with a p-value. A false discovery is a gene on that list that is not differentially expressed, i.e. a gene for which 1 = 2 , i.e. the null hypothesis for the t-test is true. Including that gene on our list of significant genes is a mistake. The FDR rate is defined as FDR = # false discoveries # significant genes
The FDR rate will depend on the decision procedure. Using a p value of < 0.0001 to declare significance should result in a smaller FDR than using a p of < 0.15. Calculating FDR for any specific study is not possible, because you don't know which genes are false discoveries which and are not. However, Benjamini and Hochberg derived a procedure that on average controls FDR to any specified value. By "controls", I mean that the specified FDR is conservative. The average FDR may be less than the specified value. The B&H FDR procedure: Specify an acceptable false discovery rate: . Calculate p values for each of n genes individually and rank the p values from smallest to largest. The smallest is denoted p(1) . The largest is p(n) . For each, compute a(i) = p(i) n/i. Find the largest i for which a(i) < . Declare as 'significant' all genes corresponding to p(1) through p(i) . Note: you may declare gene (3) significant even if a(3) > if there is a 'higher' gene, e.g. (4), with a(4) < . See the example. Example: Consider testing 10 genes. The 2-sided p-values for testing Ho: t = c for each gene separately are: A: ...