Of ANalysis VAriance (ANOVA) for a sequence of models
Model comparison can be generalized to a sequence of models (not just one full and one reduced model) Context: usual nGM model: y = X + , Let X1 = 1 and Xm = X. But now, we have a sequence of models "in between" 1 and X Suppose X2 , . . . , Xm-1 are design matrices satisfying C(X1 ) < C(X2 ) < . . . < C(Xm-1 ) < C(Xm ). We'll also define Xm+1 = I N(0, 2 I)
c 2011 Dept. of Statistics (Iowa State University)
Stat 511, section 13
1 / 19
Some examples
Multiple Regression
X1 = 1, X2 = [1, x1 ], X3 = [1, x1 , x2 ], . . . Xm = [11 x1 , . . . , xm-1 ]. SS(j + 1 | j) is the decrease in SSE that results when the explanatory variable xi is added to a model containing 1, x1 , . . . , xj-1 .
Test for linear trend and test for lack of linear fit. 1 1 1 1 1 1 1 1 1 1 2 0 1 , X2 = 1 2 , X3 = 0 X1 = 1 3 0 1 1 1 3 0 1 1 4 0 1 1 4 0
0 0 1 1 0 0 0 0
0 0 0 0 1 1 0 0
0 0 0 0 0 0 1 1
c 2011 Dept. of Statistics (Iowa State University)
Stat 511, section 13
2 / 19
Context for linear lack of fit
Let i = mean surface smoothness for a piece of metal ground for i minutes (i = 1, 2, 3, 4). MS(2 | 1) / MSE can be used to test
Ho : 1 = 2 = 3 i = 0 i = 1, 2, 3, 4 for some o IR vs. HA : i = 0 + 1 i i = 1, 2, 3, 4 for some o IR, 1 IR\{0}. This is the F test for a linear trend, 1 = 0 vs. 1 = 0
MSE(3 | 2) / MSE can be used to test
Ho : i = 0 + 1 i i = 1, 2, 3, 4 for some o 1 IR vs. HA : There does not exist 0 , 1 IR such that i = 0 + 1 i i = 1, 2, 3, 4. This is known as the F test for lack of linear fit. Compares fit of linear regression model C(X2) to fit of means model C(X3)
c 2011 Dept. of Statistics (Iowa State University)
Stat 511, section 13
3 / 19
All tests can be written as full vs. reduced model tests Which means they could be written as tests of C = d But, what is C? Especially when interpretation of changes from model to model Example:
Yi = 0 + 1 Xi + i slope is 1 Yi = 0 + 1 Xi + 2 Xi2 + i slope at Xi is 1 + 22 Xi In grinding study, Xi = 0 outside range of Xi in data
What can we say about the collection of tests in the ANOVA table?
c 2011 Dept. of Statistics (Iowa State University)
Stat 511, section 13
4 / 19
General framework
Context: usual nGM model: y = X + , Let X1 = 1 and Xm = X. Suppose X2 , . . . , Xm-1 are design matrices satisfying C(X1 ) < C(X2 ) < . . . < C(Xm-1 ) < C(Xm ). We'll also define Xm+1 = I Let Pj = PXj
n
N(0, 2 I)
j = 1, . . . , m + 1. Then
(yi - .)2 = y (I - P1 )y = y (Pm+1 - P1 )y y
i=1
= y (Pm+1 - Pm + Pm - Pm-1 + + P2 - P1 )y = y (Pm+1 - Pm )y + . . . + y (P2 - P1 )y
m
=
j=1
y (Pj+1 - Pj )y.
c 2011 Dept. of Statistics (Iowa State University)
Stat 511, section 13
5 / 19
Sequential SS
The sums of squares in the equation
m
y (I - P1 )y =
j=1
y (Pj+1 - Pj )y
are often arranged in an ANOVA table. Sum of Squares y (P2 - P1 )y y (P3 - P2 )y . . . y (Pm - Pm-1 )y y (Pm+1 - Pm )y y (I - P1 )y
c 2011 Dept. of Statistics (Iowa State University)
SS(2 | 1) SS(3 | 2) . . . SS(m | m - 1) SSE = y (I - PX )y
n
SSTot =
i=1
(yi - .)2 y
Stat 511, section 13 6 / 19
What can we say about each SS in the ANOVA table? 1) Each is a quadratic form, W AW, where W N(X, 2 I) 2) Each is proportional to a Chi-square distribution, because j = 1, . . . , m, A = (Pj+1 - Pj )/ 2 2 I is idempotent (Pj+1 - Pj )(Pj+1 - Pj ) = Pj+1 Pj+1 - Pj+1 Pj - Pj Pj+1 + Pj Pj = Pj+1 - Pj - Pj + Pj = Pj+1 - Pj . So, (slide 15 of section 8) y
(Pj+1 -Pj ) y 2
2
2(ncp)
with
ncp = X (Pj+1 - Pj )X ) d.f. = = rank(Xj+1 ) - rank(Xj )
for all, j = 1, . . . , m
c 2011 Dept. of Statistics (Iowa State University)
Stat 511, section 13
7 / 19
3) Each SS is independent, using same argument as on slide 10 of section 11 y Ay independent of y By if AB = 0 i.e. j < (Pj+1 - Pj )(P
+1
- P ) = Pj+1 P +1 - Pj+1 P - Pj P = Pj+1 - Pj+1 - Pj + Pj = 0.
+1
+ Pj P
It follows that the m 2 random variables are all independent. This result sometimes called Cochran's theorem
c 2011 Dept. of Statistics (Iowa State University)
Stat 511, section 13
8 / 19
4) Can add sequential SS. If it makes sense to test: full model X4 vs reduced model X2 , SS for that test = SS(4 | 3 ) + SS(3 | 2) = y (P4 - P3 )y + y (P3 - P2 )y In general, 3) and 4) only true for sequential SS (type I SS) Applies to other SS (e.g. partial = type III SS) only when appropriate parts of X are orthogonal to each other For factor effects models, only when design is balanced (equal # obs. per treatment)
c 2011 Dept. of Statistics (Iowa State University)
Stat 511, section 13
9 / 19
Connection to full vs. reduced SS
Each comparison of models is equivalent to a full vs. reduced model comparison: To see this, note that: SS(j + l | j) = = = = = y (Pj+1 - Pj )y y (Pj+1 - Pj + I - I)y y (I - Pj - I + Pj+1 )y y (I - Pj )y - y (I - Pj+1 )y SSEREDUCED - SSEFULL
For each test j, Hoj is E(y) C(Xj ), Ha is E(y) C(Xj+1 )
c 2011 Dept. of Statistics (Iowa State University)
Stat 511, section 13
10 / 19
F tests for sequential SS
For each sequential hypothesis, j = 1, . . . , m - 1 we have Fj = y (Pj+1 - Pj )y/[rank(Xj+1 ) - rank(Xj )] y (I - PX )y /[n - rank(X)] ncp F1 ,2 where
ncp = X (Pj+1 - P)j)X/ 2 ), and 1 = rank(Xj+1 ) - rank(Xj ) 2 = n - rank(X) define MS (j + 1 | j) =
y (Pj+1 -Pj )y
rank(Xj+1 )=rank(Xj ) Fj = MS (j + 1 | j) / MSE Under Hoj , noncentrality parameter for test j = 2011 0
c Dept. of Statistics (Iowa State University)
Stat 511, section 13
11 / 19
Details of non-centrality parameter
The noncentrality parameter is X (Pj+1 - Pj )X/ 2 =
X (Pj+1 -Pj ) (Pj+1 -Pj )X 2
= || (Pj+1 - Pj )X ||2 / 2 = || Pj+1 E(y) - Pj E(y) ||2 / 2 . If Hoj is true, Pj+1 E(y) = Pj E(y) = E(y). Thus, the ncp. for test j = 0 under Hoj .
c 2011 Dept. of Statistics (Iowa State University)
Stat 511, section 13
12 / 19
Return to issues in examples
1
2
How does X matrix for yi = 0 + 1 Xi + i relate to X matrix for yij = i + ij ? What sort of C test corresponds to this model comparison? How to interpret tests when "meaning" of 1 changes between Yi = 0 + 1 Xi + i and Yi = 0 + 1 Xi + 2 Xi2 + i ? Example: Xi {1, 2, 3, 4}. Consider 3 X matrices 1 1 0 0 0 0 0 0 X1 0 0 0 0 1 0 1 0 0 1 0 1 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 X2 1 1 1 1 1 1 2 4 8 2 4 8 3 9 27 3 9 27 4 16 64 4 16 64 X3 1 1 1 1 1 1 1 1 -3 -3 -1 -1 1 1 3 3 1 1 -1 -1 -1 -1 1 1 -1 -1 3 3 -3 -3 1 1
c 2011 Dept. of Statistics (Iowa State1 University)
My claim: C(X ) = C(X2 ) = C(X3 )
Stat 511, section 13
13 / 19
Columns of X2 are X's in a cubic regression: Yi = 0 + 1 Xi + 2 Xi2 + 3 Xi3 + i A cubic perfectly fits four points, so C(X2 ) = C(X1 ) So comparison of Yij = 0 + 1 Xij + ij vs Yij = i + comparison of Yij = 0 + 1 Xij + ij vs 2 3 Yij = 0 + 1 Xij + 2 Xij + 3 Xij + ij Now very clear that C(Xo ) C(X1 ) = C(X2 ) Model comparison test same as C test of 2 = 0 and 3 = 0.
ij
is same as
c 2011 Dept. of Statistics (Iowa State University)
Stat 511, section 13
14 / 19
Each column of X3 can be expressed in terms of columns of X2 Define Xi [j] as the j'th column of Xi
X3 [2] = 2X2 [2] - 5X3 [1] X3 [3] = 2(X2 [3] - X3 [2] + 7.5)/5 X3 [4] = 10(X2 [4] - 7.5X3 [3] - 10.4X3 [2] - 25)/3
Why consider X3 ? 2 0 0 0 X1 X1 0 0 2 0 0 2 0 0 X2 X2 0 4 10 30 100 10 30 100 354 0 0 30 100 354 1300 2 100 354 1300 4890 X3 X3 4 0 0 0 0 20 0 0 0 0 4 0 0 0 0 20
c 2011 Dept. of Statistics (Iowa State University)
Stat 511, section 13
15 / 19
Orthogonal polynomials
Columns of X3 are orthogonal, when sample sizes equal estimates of 's are independent ((X3 X3 )-1 is diagonal). Columns of X3 are one example of a set of orthogonal polynomials. Uses of orthogonal polynomials:
Historical: fitting a regression. (X X)-1 much easier to compute Analysis of quantitative ("amount of") treatments: Decompose SS for trt into additive components due to linear, quadratic... Extends to interactions, e.g. linear A x linear B Alternate basis for full-rank parameterization (instead of drop first) Numerical stability for regressions
c 2011 Dept. of Statistics (Iowa State University)
Stat 511, section 13
16 / 19
Orthogonal polynomials - 2
I once tried to fit a cubic regression, X = year: 1992, 1993, ... 2006 software complained: X matrix not full rank, X3 dropped from model Correlation matrix of estimates, ((X X)-1 scaled so 1's on diagonal, when X = 1, 1, 2, 2, 3, 3, 4, 4 1.0000000 -0.9871142 0.9652342 -0.9421683 -0.9871142 1.0000000 -0.9934490 0.9798135 0.9652342 -0.9934490 1.0000000 -0.9960238 -0.9421683 0.9798135 -0.9960238 1.0000000 Correlations even closer to 1 for X = 1992 ... 2006 that X X matrix fails numerical test for singularity for fun, plot X 2 vs. X 3 or X 3 vs. X 4
c 2011 Dept. of Statistics (Iowa State University) Stat 511, section 13 17 / 19
Orthogonal polynomils - 3
Consequence is numerical instability in all computations How can we reduce correlations among columns in X matrix?
1
Center X's at mean X. Xi = Xi - X Correlation matrix of estimates, ((X X)-1 scaled so 1's on diagonal, when X = 1, 1, 2, 2, 3, 3, 4, 4 1.0000000 0.0000000 -0.7808688 0.0000000 0.0000000 1.0000000 0.0000000 -0.9597374 -0.7808688 0.0000000 1.0000000 0.0000000 0.0000000 -0.9597374 0.0000000 1.0000000 Correlations much reduced, still have correlations between odd powers and between even powers
Use orthogonal polynomials: all estimates uncorrelated
c 2011 Dept. of Statistics (Iowa State University)
Stat 511, section 13
18 / 19
Coefficients for orthogonal polynomials
Where do you find coefficients? Tables for statisticians, e.g. Biometrika tables, vol. 1 Only for equally spaced X's, equal numbers of obs. per X Remember the vector space discussion from week 1
n obs. Xi is vector of X i .
C0 = 0'th degree orthog. poly. is a vector of 1's = X0 . linear orthog. poly.: want to find C1 so that C1 orthog. to X0
X1 is a point in n-dimensional space C(C0 ) is a subspace. Want to find a basis vector for the subspace C(C0 ). That is (I - PC0 )X1 , i.e. residuals from regression of X1 on C0
linear coeff: proportional to residuals of regr. X1 on C0 quadratic coeff. are residuals from regr. of X2 on [C0 , C1 ] Ci is prop. to residuals from regr. of Xi on [C0 , C1 , . . . Ci-1 ]
c 2011 Dept. of Statistics (Iowa State University) Stat 511, section 13 19 / 19
Summary
Given a sequence of design matrices, X1 , X2 , . . . Xm with C(X1 ) C(X2 ) . . . C(Xm ), the sequential SS, SS(2 | 1), SS(3 | 2), . . . SS(m | m-1) are proportional to independent 2 random variables Provides a useful way to think about decomposing a SS into components. i.e. in ANOVA lack of fit test Orthogonal polynomials provide a set of basis functions for differences in means. Most useful for quantitative treatments.
c 2011 Dept. of Statistics (Iowa State University)
Stat 511, section 13
20 / 19