Experimental Design
Let’s assume you are testing three attributes for a direct mail piece (font, color and tone). Let’s also assume each attribute has two values (Times New Roman vs. Arial for font; blue vs. black for color; and friendly vs. professional for tone). In this test, there are a total of 8 =23 = 2 x 2 x 2 combinations to test.
If you were testing test seven attributes, the number of combinations explodes to 128 =27 = 2 x 2 x 2 x 2 x 2 x 2 x 2. Consequently, a full factorial experimental design would require128 different cells of unique creative. Since there are minimum sample sizes required to accurately measure each combination, a full factorial design quickly becomes cost prohibitive.
We can test the same number of attributes with dramatically fewer cells by using fractional factorial experimental designs. In these designs only a specific subset of treatment combinations are needed. For example, instead of needing 8 cells to test 3 attributes, we can accomplish much of the same in 4. The impact of these designs is more dramatic when more features are tested. A combination of 7 features can be tested in 8 combinations rather than 27=128.
There are tradeoffs with these designs. You will get a good read on the impact of individual attributes in many contexts, but you will not be able to measure the impact of all combinations. Nevertheless, the cost and efficiency of these tests make them very attractive.
Fractional factorial designs can be found in the NIST’s Engineering Statistics Handbook.