There are three overarching philosophies to conducting MVTs with online proper

# There are three overarching philosophies to

• 42

This preview shows page 21 - 23 out of 42 pages.

There are three overarching philosophies to conducting MVTs with online proper- ties. 4.1 Traditional MVT This approach uses designs that are used in manufacturing and other offline applica- tions. These designs are most often fractional factorial ( Davies and Hay 1950 ) and Plackett and Burman ( 1946 ) designs that are specific subsets of full factorial designs (all combinations of factor levels). These designs were popularized by Genichi Taguchi and are sometimes known as Taguchi designs. The user must be careful to choose a design that will have sufficient resolution to estimate the main effects and interactions that are of interest. For our MSN example we show designs for a test of these five factors with a full factorial, a fractional factorial or a Plackett-Burman design. 123
Controlled experiments on the web 161 Table 1 Fractional factorial design to test five factors with eight user groups User groups Factor levels assigned to each group F1 F2 F3 F4 F5 1 1 1 1 1 1 2 1 1 1 1 1 3 1 1 1 1 1 4 1 1 1 1 1 5 1 1 1 1 1 6 1 1 1 1 1 7 1 1 1 1 1 8 1 1 1 1 1 Full factorial has all combinations of the factors which would be 2 5 = 32 user groups. A fractional factorial is a fraction of the full factorial that has 2 K user groups and each column is orthogonal to the other four columns. There are obviously many such fractions with 8 and 16 user groups. One fractional factorial for K = 3 is given in Table 1 where 1 denotes the control and 1 denotes the treatment. Plackett–Burman designs can be constructed where the factors are all at two levels with the number of user groups being a multiple of 4, so 4, 8, 12, 16, 20, etc. The number of factors that can be tested for any of these designs is the number of user groups minus one. If the number of user groups is a power of two the Plackett–Burman design is also a fractional factorial. As with the fractional factorials, there are usually many Plackett–Burman designs that could be used for a given number of user groups. In the statistical field of Design of Experiments, a major research area is to find designs that minimize the number of user groups needed for the test while allowing you to estimate the main effects and interactions with little or no confounding. The fractional factorial in Table 1 can estimate all five main effects but cannot estimate interactions well ( Box et al. 2005 , pp. 235–305). For many experimenters one of the primary reasons for running an MVT is to estimate the interactions among the fac- tors being tested. You cannot estimate any interactions well with this design since all interactions are totally confounded with main effects or other two-factor interac- tions. No amount of effort at analysis or data mining will allow you to estimate these interactions individually. If you want to estimate all two factor interactions with five factors you will need a fractional factorial design with 16 treatment combinations. The Placket–Burman design in Table 2