/ Randomization Permutation tests
Bootstrapping preserves the fixed effects
Difference of two means: resample Y1i and resample Y2i bootstrap estimates, 1B - 2B , are centered on/near Y1 - Y2 , ^ ^ which estimates 1 - 2 Regression bootstrap: ^ resample ^i , add to X ^ ^ B centered on/near , which estimates
so a bootstrap provides a confidence interval for tests of Ho: = k are indirect, via inclusion of k in 1 - confidence interval Sometimes said: "bootstrapping resamples data under Ha".
c 2011 Dept. Statistics (Iowa State University)
Stat 511 section 15.5
1/9
Randomization/permutation tests
A randomization / permutation test resamples the data under Ho. used to test Ho without making many assumptions Example: E Yij = i , i = 1, 2, want to test Ho: 1 - 2 = 0. A permutation test would:
define a test statistic, e.g. T = Y1 - Y2 calculate To from the observed data pool all observations into a single group: {Y1j , Y2j } permute group labels over the pooled observations maintains n1 and n2 calculate TRi from the relabelled observations repeat for all possible permutations of labels FTR is the permutation distribution of the test statistic If To > 0, the two-sided p-value is P[TR -To ] + P[TR To ]
c 2011 Dept. Statistics (Iowa State University)
Stat 511 section 15.5
2/9
Randomization/permutation tests
Resampling from a single pool of observations tests HoR : F1 (x) = F2 (x) Notice a subtle point: HoR is slightly more general than Ho:1 = 2 H0R is the same as Ho if you assume the two populations have same distribution (e.g. normal) and same variance This subtlety is usually ignored in practice Names:
Permutation test: enumerate all possible permutations Randomization test: enumerate a sample of permutations
c 2011 Dept. Statistics (Iowa State University)
Stat 511 section 15.5
3/9
Permutation tests
Some sophisticated algorithms available to enumerate all possible permutations. Results sometimes called "exact" tests Cyrus Mehta responsible for a lot of the algorithm development software called StatXact, algorithms also now in SAS But often # permutations too large to enumerate
c 2011 Dept. Statistics (Iowa State University)
Stat 511 section 15.5
4/9
Randomization tests
Randomization tests: random sample (with replacement) of all possible permutations one subtlety:
Observed data is one possible permutation Should include obs. data with random sample so p-value is (Y + 1)/(N + 1), where Y is the number of random samples with TR - | To | or TR | To | N is total the number of random samples has the nice properties of avoiding p = 0, and if To is extreme, increasing N decreases p
c 2011 Dept. Statistics (Iowa State University)
Stat 511 section 15.5
5/9
Nonparametric tests
The standard nonparametric tests (Wilcoxon rank sum, Wilcoxon signed rank, Page, Kruskal-Wallis, Friedman, Spearman rank) are all permutation tests When data reduced to ranks, the permutation distribution given n1 , n2 is the same for any data set. Enumerate the distribution once, tabulate, and use for any data set. Tabulated p-values for these tests are based on the exact permutation distribution Commonly find a shift to asymptotic inference (normal approximations) for larger n's. Moment calculations based on permutation of ranks among observations.
c 2011 Dept. Statistics (Iowa State University)
Stat 511 section 15.5
6/9
Practical aspects
Permutation / Randomization must respect the study design Completely randomized design: permute treatment labels Block design: permute treatment labels within blocks Paired data: retain labels [p=0.5] or switch labels [p=0.5] Repeated measures: permute treatment labels, but retain vector of repeated observations together. Need to think about how to test time effects. Regression: permute Y values. Retains structure of X's.
c 2011 Dept. Statistics (Iowa State University)
Stat 511 section 15.5
7/9
Aside on expt. design and bootstrap
What about bootstrapping data from something other than a CRD? What are exchangeable? Usually the residuals Use regression bootstrap, where some of the terms account for the expt. design for data from an RCBD
fit model, e.g. yij = + i + j + ij , estimate {^ij }, i.e. the residuals resample the residuals: B form bootstrap sample yBij = ^ij + eBij y estimate C of interest from yB .
c 2011 Dept. Statistics (Iowa State University)
Stat 511 section 15.5
8/9
Practical aspects
Randomization / Permutation tests easiest for simple hypotheses: F1 (x) = F2 (x) = F3 (x). Harder to test non-simple Ho, e.g. two-factorial, test HoA : no A main effect want to generate randomizations where HoA true, but not restricting the data otherwise Simple randomization generates data for which HoA is true, and HoB : no B main effect, and HoAB : no A*B interaction are true Recent progress (work and books by Pesarin) How many randomizations: use Binomial theory
Y Bin(N, ) if = 0.05, se ^ = se Y/N 0.22/ N. p N = 1, 000, se 0.69 N = 10, 000, se 0.22
c 2011 Dept. Statistics (Iowa State University) Stat 511 section 15.5 9/9