There are three overarching philosophies to conducting MVTs with online proper-
ties.
4.1 Traditional MVT
This approach uses designs that are used in manufacturing and other offline applica-
tions. These designs are most often fractional factorial (
Davies and Hay 1950
) and
Plackett and Burman
(
1946
) designs that are specific subsets of full factorial designs
(all combinations of factor levels). These designs were popularized by Genichi Taguchi
and are sometimes known as Taguchi designs. The user must be careful to choose a
design that will have sufficient resolution to estimate the main effects and interactions
that are of interest.
For our MSN example we show designs for a test of these five factors with a full
factorial, a fractional factorial or a Plackett-Burman design.
123
Controlled experiments on the web
161
Table 1
Fractional factorial
design to test five factors with
eight user groups
User groups
Factor levels assigned to each group
F1
F2
F3
F4
F5
1
−
1
−
1
−
1
1
1
2
−
1
−
1
1
1
−
1
3
−
1
1
−
1
−
1
1
4
−
1
1
1
−
1
−
1
5
1
−
1
−
1
−
1
−
1
6
1
−
1
1
−
1
1
7
1
1
−
1
1
−
1
8
1
1
1
1
1
Full factorial has all combinations of the factors which would be 2
5
=
32 user
groups. A fractional factorial is a fraction of the full factorial that has 2
K
user groups
and each column is orthogonal to the other four columns. There are obviously many
such fractions with 8 and 16 user groups. One fractional factorial for
K
=
3 is given
in Table
1
where
−
1 denotes the control and 1 denotes the treatment.
Plackett–Burman designs can be constructed where the factors are all at two levels
with the number of user groups being a multiple of 4, so 4, 8, 12, 16, 20, etc. The
number of factors that can be tested for any of these designs is the number of user
groups minus one. If the number of user groups is a power of two the Plackett–Burman
design is also a fractional factorial.
As with the fractional factorials, there are usually many Plackett–Burman designs
that could be used for a given number of user groups.
In the statistical field of Design of Experiments, a major research area is to find
designs that minimize the number of user groups needed for the test while allowing
you to estimate the main effects and interactions with little or no confounding. The
fractional factorial in Table
1
can estimate all five main effects but cannot estimate
interactions well (
Box et al. 2005
, pp. 235–305). For many experimenters one of the
primary reasons for running an MVT is to estimate the interactions among the fac-
tors being tested. You cannot estimate any interactions well with this design since
all interactions are totally confounded with main effects or other two-factor interac-
tions. No amount of effort at analysis or data mining will allow you to estimate these
interactions individually. If you want to estimate all two factor interactions with five
factors you will need a fractional factorial design with 16 treatment combinations. The
Placket–Burman design in Table
2
