mis-specification Model and confounding
Suppose we have a data generating model of the form
Y = + X + Z + . The usual assumptions E( |X, Z) = 0 and var ( |X, Z) = 2 hold. The covariate X is observed, but Z is not observable. If we regress Y on X, the model we are fitting differs from the data generating model. What are the implications of this? Does the fitted regression model reflect E(Y |X) and var(Y |X)?
2
^ The simplest case is where X and Z are independent. The slope estimate has the form ^ =
i
Yi(Xi - X)/
i
(Xi - X)2 (Xi - X)2
i i (Xi i
=
i
( + Xi + Zi + i)(Xi - X)/ Zi(Xi - X)/
i i
= +
(Xi - X)2 +
- X)/
i
(Xi - X)2
By the double expectation theorem, E( |X) = EZ|X E( |X, Z) = 0 and since Z and X are independent E(Zi(Xi - X)|X) =
i i
(Xi - X)E(Zi|X) = EZ
i
(Xi - X) = 0.
3
^ Therefore remains unbiased if there is an unmeasured covariate Z that is independent of X. What about 2 ? ^
4
The residuals are
(I - P )Y = (I - P )(Z + ) So the residual sum of squares is
Y (I - P )Y
= 2 Z (I - P )Z + (I - P ) + 2Z (I - P ) .
The expected value is therefore
EY (I - P )Y
= 2 var(Z)rank(I - P ) + 2 rank(I - P ) = ( 2 var(Z) + 2 )(n - 2).
Hence the MSE (^2 ) has expected value 2 var(Z) + 2 .
5
Are our inferences correct? Since cov(Z + |X) = ( 2 var(Z) + 2 )I,
E(Z + |X) = 0
we can view Z + as being the error term and all the results for correctly specified models hold.
6
Confounding Continue to assume the data generating model
Y = + X + Z + , but now suppose that X and Z are correlated. As before, Z is not observed so our analysis will be based on Y and X. A variable such as Z that is associated with both the dependent and independent variables in a regression model is called a "confounder."
7
Interpretation of non-orthogonal covariate effects Suppose X and Z are standardized, and cor(X, Z) = r. If X increases by one unit and Z remains fixed, the expected response increases by units. If Z increases by one unit and X remains fixed, the expected response increases by units. This interpretation is relevant if we can control the value of X or Z without changing the other value. However, if we select from our sample a pair of individuals with X values differing by one unit, their Z values will tend to differ by r units. Therefore these two individuals will have expected responses differing by + r units.
8
Suppose our interest lies mainly with X. A known confounder Z that can be measured and included in a regression model does not generally pose a problem unless it is highly collinear with X. Unknown and/or unmeasured confounders place major limits on our ability to interpret regression models causally or mechanistically. The best way to be sure that confounding is not an issue is to randomly assign the levels of X. In this case, there can be no association between X and Z.
9
For simplicity, suppose that Z has mean 0 and variance 1, and we use least squares to fit a working model
^ ^ Y = + X ^ We can work out the limiting value of the slope estimate as follows.
^ =
- X) 2 i (Xi - X) i ( + Xi + Zi + i )(Xi - X)/n = 2 i (Xi - X) /n + r.
i Yi (Xi
Note that if either = 0 (Z is independent of Y given X) or if r = 0 (Z is independent of X), then is estimated correctly.
10
Since
^ + r, and it is easy to show that , the fitted model is approximately ^
^ Y + X + rX. How does the fitted model relate to E(Y |X)? It is easy to get
E(Y |X) = + X + E(Z|X), so the fitted regression model agrees with E(Y |X) as long as
E(Z|X) = rX.
11
Turning now to the variance structure of the fitted model, the limiting value of 2 is ^
2 ^
=
i
(Yi - - Xi)2 /(n - 2) ^ ^ (Zi +
i 2 i
- rXi)2 /n
+ 2 (1 - r2 ). Ideally this should estimate var(Y |X).
12
By the law of total variation,
var(Y |X) = EZ|X var(Y |X, Z) + varZ|X E(Y |X, Z) = 2 + varZ|X ( + X + Z) = 2 + 2 var(Z|X). So for 2 to estimate var(Y |X) we need ^ var(Z|X) = 1 - r2 .
13
Gaussian case Suppose
Y =
A B
is a Gaussian random vector, where Y Rn, A Rq , and B Rn-q . Let = EY and = cov(Y ). We can partition and as
=
1 2
=
11 12 21 22
,
where 1 Rq , 2 Rn-q , 11 Rqq , 12 Rqn-q , 22 Rn-qn-q , and 21 = 12 .
14
Gaussian case (continued) It is a fact that A|B is Gaussian with mean E(A|B) = 1 + 12 -1 (B - 2 ) 22 and covariance matrix cov(A|B) = 11 - 12 -1 21 . 22
15
Now we apply these results to our model
X = Z + , assuming Z and to be jointly Gaussian. The mean vector and covariance matrix are
E we so get
Z X
=0
cov
Z X
=
1 r r 1
.
E(Z|X) = rX
cov(Z|X) = 1 - r2 .
These are exactly the conditions stated above that guarantee the fitted mean model converges to E(Y |X) and the fitted variance model converges to var(Y |X).
16
Consequences of confounding How does the presence of unmeasured confounders affect our ability to interpret regression models?
17
Population average covariate effect Suppose we specify a value X in the covariate space and randomly select two subjects i and j having X values Xi = X +1 and Xj = X . The inter-individual difference is
Yi - Yj = + (Zi - Zj ) + which has a mean value of
i
-
j,
E(Yi - Yj |Xi = X + 1, Xj = X ) = + (E(Z|X = X + 1) - E(Z|X = X )), which agrees with what would be obtained by least squares analysis as long as E(Z|X) = rX.
18
The variance of Yi - Yj is 2 2 + 2 2 var(Z|X), which also agrees with the results of least squares analysis as long as var(Z|X) = 1 - r2.
19
Individual treatment effect Now suppose we match two subjects i and j having X values differing by one unit, and who also having the same values of Z. This is what one expect to see as the pre-treatment and post-treatment measurements following a treatment that changes an individual's X value by one unit, if the treatment does not affect Z. The mean difference (individual treatment effect) is E(Yi - Yj |Xi = X + 1, Xj = X , Zi = Zj ) = and the variance is var(Yi - Yj |Xi = X + 1, Xj = X , Zi = Zj ) = 2 2 . These do not agree with the estimates obtained by using least squares to analyze the observable Y, X data. Depending on the sign of , we may either overstate or understate the individual treatment effect , and the population variance of the treatment effect will always be overstated.
20
Errors in Variables for Linear Models Suppose the data generating model is
Y = Z + , with the usual linear model assumptions, but we do not observe Z. Rather, we observe
X = Z + , where is a random vector of covariate measurement errors with E = 0. Assuming X1 = 1 is the intercept, it is natural to set 1 0. This is called an "errors in variables" model, or a "measurement error model."
21
When covariates are measured with error, least squares point estimates may be biased and inferences may be incorrect. Intuitively it seems that slope estimates should be "attenuated" (biased toward zero). The reasoning is that as the measurement error grows very large, the observed covariate X becomes equivalent to noise, so the slope estimate should go to zero.
22
Let X and Z now represent the n p + 1 observed and ideal design matrices. The least squares estimate of the model coefficients is
^ = (X X)-1 X Y = (Z Z + Z + Z + )-1 (Z Y + Y ) = (Z Z/n + Z /n + Z/n + /n)-1 (Z Y /n + Z/n + /n). We will make the simplifying assumption that the covariate measurement error is uncorrelated with the covariate levels, so
Z /n 0, and that the covariate measurement error and observation error correlated, so are un-
/n 0.
23
Under these circumstances, ^ (Z Z/n + /n)-1 Z Y /n. Let Mz be the limiting value of Z Z/n, and let M be the limiting value of ^ /n. Thus the limit of is
(Mz + M )-1 Z Y /n
-1 -1 = (I + Mz M )-1 Mz Z Y /n -1 (I + Mz M )-1 0 .
and hence the limiting bias is
-1 0 - = ((I + Mz M )-1 - I).
24
What can we say about the bias?
-1 Note that the matrix Mz M has non-negative eigenvalues, since it shares its eigenvalues with the positive semi-definite matrix
-1/2 -T Mz M Mz /2 .
-1 It follows that all eigenvalues of I + Mz M are greater than or equal to 1, -1 so all eigenvalues of (I + Mz M )-1 are less than or equal to 1. -1 This means that (I + Mz M )-1 is a contraction, so 0 .
Therefore the sum of squares of fitted slopes is smaller on average than the sum of squares of actual slopes, due to measurement error.
25
SIMEX SIMEX (simulation-extrapolation) is a relatively straightforward way to adjust for the effects of measurement error if the variances and covariances among the measurement errors can be considered known. Regress Y on X + E, where E is simulated noise having the same variance as the assumed measurement error. Denote the coefficient vector of this fit ^ as . ^ Repeat this for several values of 0, leading to a set of vectors. ^ Ideally, -1 would approximate the coefficient estimates under no measurement error. ^ By fitting a line or smooth curve to the values (separately for each com^ ponent of ), it becomes possible to extrapolate back to -1 .
26