In Introduction to the Design and Analysis of Experiments, George Cobb (1998) describes the variability inherent an
experiment in the following way:
Any experiment is likely to involve three kinds of variability:
Planned, systematic variability.
This is the kind we want since it includes the differences due to the treatments.
This is the kind our probability models allow us to live with.
We can estimate the size of this
variability if we plan our experiment correctly.
Unplanned, systematic variability.
This kind threatens disaster! We deal with this variability in two ways, by
randomization and by blocking.
Randomization turns unplanned, systematic variation into planned, chance-like
variation, while blocking turns unplanned, systematic variation into planned, systematic variation.
The management of these three sources of variation is the essence of experimental design.
The management of the unplanned, systematic variability (3) is the issue between you and Peter.
One important goal of
randomization is to turn this systematic variability into chance-like variation that adds into our standard error.
randomization "fails", then all bets are off.
Randomization is never a guarantee against confounding variables, but it is our
This is an example I created to try to clarify the situation.
(Linda Young bears no responsibility in what follows, but,
nevertheless, it is all her fault.)
To focus the discussion, consider the variation inherent in the following experimental setting.
To keep the computations
simple and clear, the sample size is unrealistically small.
I hope the gain in simplicity and clarity from this example
outweighs the obvious problem with sample size.
Compare two kinds of rabbit food on weight gain (in ounces) from the age of two weeks to the age of six weeks of
We want to know if the rabbits will gain more weight on one diet than on the other.
We have space to house eight
rabbits for this experiment.
Sources of Variation:
The most obvious is, perhaps, that the rabbits are all different rabbits, and so they will all grow at different rates.
different breeds of rabbit are used, then we will have an additional source of variation.
Young California rabbits do not grow
at the same rate as young Florida White rabbits, for example.
The environment in which the rabbits live will not be exactly
They cannot all live in the same location; some will be in slightly warmer areas while others will be in areas with
They will not all have exactly the same amount of exercise or sleep.
The food will be carefully weighed before it
is given to the rabbits, but there will inevitably be measurement error in the amount of food given to each rabbit.
the rabbits will be weighed before the experiment begins and after the experiment ends.
Measurement error (hopefully small)
will occur in both these weighings.
All of the aspects of the experimental setting mentioned so far can be considered natural