Common Mistakes in Experimentation

1. Variation due to experimental error is ignored.

   Every measured value is a random variable.

   So compare variation due to a factor with variation due to errors!

2. Important parameters are not controlled.

   So identify all factors, then select a subset to vary!

   Accounting for the effect of users is particularly important.

3. Effects of different factors are not isolated.

   You may not be able to figure out which factor caused a response variable change.

   So try not to vary several factors simultaneously.

4. Simple one-factor-at a time designs are used.

   Could you use another experiment design to obtain narrower confidence intervals with the same number of experiments?

5. Interactions are ignored.

   One-factor-at-a-time designs cannot estimate interactions between factors. For example, the effect on performance of adding a 1 Kbyte cache may depend on the program size.

   So don't use a one-factor-at-a-time design!

6. Too many experiments are conducted.

   So use a sequence of smaller experiment designs that idsolate critical factors, rather than one experiment with a zillion factors!

Navigation: Previous, Up, Next