whats So next?
!"#!$%&%'
()*)$+,-.)//01*$2*345/0/
607).)*834$+,-.)//01*$99
1.Hierarchical/Bayesian methods to
estimate variance (and Empirical
Bayesian ways to produce estimates)
2.Non-parametric ways of obtaining pvalues
3.Corrections for multiple testing
A bag of tricks for the large data analyst!
Thursday, February 17, 2011
Thursday, February 17, 2011
Naive Bayes Classiers
Consider the hector-recognition computer
vision problem: is this a picture of hector?
Introduction to Empirical
Bayes
Input: pixel intensities (continuous) xi
Output: yes/no (categorical) g
g
x1
x2
...
xp
Decision: P(g=yes | x1, ... , xp) > P(g=no | x1, ... , xp) ?
Thursday, February 17, 2011
Thursday, February 17, 2011
Naive Bayes Classiers
Bayes Rule
Consider the hector-recognition computer
vision problem: is this a picture of hector?
P (G|X ) P (X |G)P (G)
Posterior
Decision: P(g=yes | x1, ... , xp) > P(g=no | x1, ... , xp) ?
Dening P(g=yes | x1, ... , xp) is hard
Dening P(x1, ... , xp | g=yes) is easier
g
x1
x2
...
xp
P (x1 , . . . , xp |g = yes ) =
p
j =1
f (xj |g = yes )
2
Xj |G N (j , j )
their analyses
Priors may be fully specied (full Bayesian), or its
parameters estimated from data (empirical Bayesian)
The subject of much debate in statistics (Frequentist/
Bayesian argument)
Useful way of thinking about hierarchical models
Thursday, February 17, 2011
Robust Naive Bayes Classier
Consider the hector-recognition computer
vision problem: is this a picture of hector?
Many parameters to estimate (as many
means as pixels), not that much data (there
arent as many images)
g
1
x1
2
x2
...
...
p
xp
Idea: Pixels behave similarly, can we use
pool data to estimate pixel means
j |G N (0 , 2 )
2
Xj |j N (j , j )
Thursday, February 17, 2011
Prior
Statisticians use it as a way of bringing history into
Now what?
Thursday, February 17, 2011
Likelihood
Thursday, February 17, 2011
Thursday, February 17, 2011
Thursday, February 17, 2011
Modeling Relative
Expression
Courtesy of Gordon Smyth
Thursday, February 17, 2011
Thursday, February 17, 2011
Hierarchical Model
Normal Model
Prior
Posterior Statistics
Posterior variance estimators
Moderated t-statistics
Reparametrization of Lnnstedt and Speed 2002
Normality, independence assumptions are wrong but
convenient, resulting methods are useful
Thursday, February 17, 2011
Shrinkage of Standard
Deviations
Eliminates large t-statistics merely from very small s
Thursday, February 17, 2011
Signicance analysis of
microarrays (SAM)
A clever adaptation of the t-ratio to
borrow information across genes
The data decides whether
should be closer to tg,pooled or to tg
Thursday, February 17, 2011
Thursday, February 17, 2011
The exchangeability factor
SAM-statistic
Chosen to make signal-to-noise ratios
independent of signal
Computation
For gene i
di =
Let
yi " xi
si + s0
x i = mean of Group II samples
!
!
Standard deviation of residuals for gene i
aassuming same variance
Exchangeability factor estimated using all genes
Thursday, February 17, 2011
Thursday, February 17, 2011
values. Let
values,
For
Compute
! where mad is the median absolute deviation from the
median, divided by 0.64
Compute
= coefcient of variation of the
Choose
and
Thursday, February 17, 2011
So whats next?
Scatter plots of relative difference
Random fluctuations
in the data, measured by
balanced permutations
(for cell line 1 and 2)
percentile of the
Compute the 100 quantiles of the
denoted by
y i = mean of Group I samples
!
be the
Relative difference for
a permutation of the data
that was balanced
between cell lines 1 and
2.
1.Hierarchical/Bayesian methods to
estimate variance (and Empirical
Bayesian ways to produce estimates)
2.Non-parametric ways of obtaining pvalues
3.Corrections for multiple testing
A bag of tricks for the large data analyst!
Thursday, February 17, 2011
Hypothesis testing
Once you have a given score for each gene, how do
you decide on a cut-off?
Inference and the Multiple
Comparison Problem
Many slides courtesy of John
Storey
p-values are popular.
But how do we decide on a cut-off?
Are 0.05 and 0.01 appropriate?
Are the p-values correct?
Thursday, February 17, 2011
Thursday, February 17, 2011
P-values by permutation
p-values by permutations
It is common that assumptions about
statistics used to summarize interest
(e.g., t-statistic) are not approximate
enough to yield useful p-values
We focus on one gene only. For the bth iteration, b = 1, !!! , B;
An alternative is to use permutations
After all the B permutations are done;
1. Permute the n data points for the gene (x). The rst n1 are
referred to as treatments, the second n2 as controls.
For each gene, calculate the corresponding sample
t-statistic, two tb.
1. Put p = #{b: |tb| |tobserved|}/B.
Thursday, February 17, 2011
Thursday, February 17, 2011
The reference distribution
Order the values of
(could be any stat)
Permute the treatment labels, and compute a new
set of ordered values
:3;<$=1$#2"
Repeat step 2 for, say, 100 permutations:
From these, compute the average largest,
average second largest etc.
>&
Thursday, February 17, 2011
Thursday, February 17, 2011
Selected genes
expected d(i)
Thursday, February 17, 2011
Thursday, February 17, 2011
Multiple Comparison Problem
If we do have useful approximations of
our p-values, we still face the multiple
comparison problem
Hypothesis Testing
Test for each gene null hypothesis: no
differential expression.
Two types of errors can be committed
Type I error or false positive (say that a gene is differentially
When performing many independent
tests p-values no longer have the same
interpretation
Thursday, February 17, 2011
Type II error or false negative (fail to identify a truly
differentially expressed gene, i.e.,fail to reject a false null hypothesis)
Thursday, February 17, 2011
Multiple Hypothesis Testing
What happens if we call all genes signicant
with p-values 0.05, for example?
Null True
expressed when it is not, i.e., reject a true null hypothesis).
Called
Not Called
Signicant Signicant
V
m0 V
Total
m0
Altern.True
S
m1 S
m1
Total
R
mR
m
Multiple Testing problem not
only in genomics
Statistical Comparisons of Classiers
over Multiple Datasets, Demsar, JMLR
2006
Permutation Tests for Studying
Classier Performance, Ojala, JMLR,
2010
On Comparing Classiers: Pitfalls to
avoid and a recommended approach,
Salzberg, 1997, Data Mining and
Knowledge Discovery
Null = Equivalent Expression; Alternative = Differential Expression
Thursday, February 17, 2011
Thursday, February 17, 2011
Other ways of thinking of P-values
A p-value is dened to be the minimum
false positive rate at which an observed
statistic can be called signicant
If the null hypothesis is simple, then a
null p-value is uniformly distributed
Multiple Hypothesis Test
Error Controlling Procedure
Suppose m hypotheses are tested with pvalues p1, p2, , pm
A multiple hypothesis error controlling
procedure is a function T(p; ") such that
rejecting all nulls with pi T(p; ") implies that
Error "
Error is a population quantity (not random)
Thursday, February 17, 2011
Thursday, February 17, 2011
Error Rates
Weak and Strong Control
If T(p; ") is such Error " only when m0 = m,
then the procedure provides weak control of
the error measure
If T(p; ") is such Error " for any value of m0,
then the procedure provides strong control of
the error measure note that m0 is not an
argument of T(p; ")!
Thursday, February 17, 2011
Per comparison error rate (PCER): the expected value of the number
of Type I errors over the number of hypotheses
"
PCER = E(V)/m
Per family error rate (PFER): the expected number of Type I errors
"
PFER = E(V)
Family-wise error rate: the probability of at least one Type I error
"
FEWR = Pr(V 1)
False discovery rate (FDR) rate that false discoveries occur
"
FDR = E(V/R; R>0) = E(V/R | R>0)Pr(R>0)
Positive false discovery rate (pFDR): rate that discoveries are false
"
pFDR = E(V/R | R>0)."
Thursday, February 17, 2011
FDR plug-in
Bonferroni Procedure
Create K permutations of the data,
producing statistics tjk for features
j=1,...,M and permutations k=1,...,K.
Provides strong control!..
For a range of cutoffs C, let
Robs =
M
j =1
I (|tj | > C )
E (V ) =
MK
1
I (|tk | > C )
j
K j =1
k=1
Estimate the FDR by
F DR = E (V )/Robs
Thursday, February 17, 2011
Thursday, February 17, 2011
#2"?$3@30*
expected d(i)
AB
Thursday, February 17, 2011
Thursday, February 17, 2011
Delta
Ave #
False
# called
falsely
discovery
signicant
signicant
rate
0.3
75.1
294
0.255
0.4
33.6
196
19.8
160
0.123
0.7
10.1
94
0.107
1.0
4.0
46
More than two groups
Paired data
Survival data, with censored response
0.171
0.5
More general versions of SAM
0.086
Delta is the half-width of the bar around the 45-degree line.
Thursday, February 17, 2011
Thursday, February 17, 2011
Finally
Limitations of SAM
SAM nowadays uses the q-value method
to quantify signicance
Solutions for s_0 are often at the
extremes and sensitive to the
resolution of the quantile grid.
Permutation analysis throws all
genes in the same bag
Requires a monotone signal-to-noise
relationship
we did not discuss this
There is a nice discussion of these
issues in Section 18.7 of Elements of
Statistical Learning
Thursday, February 17, 2011
Thursday, February 17, 2011