It is virtually impossible to conduct an experiment where the only difference among treatment groups is the experimental manipulation.
Example: In the human interface design,
Solutions to subject variability:
Neutralize all factors not involved in the treatments.
Decide which of the different treatments a subject will receive by some random means at the time of his or her arrival for the experiment. Do this until the desired number of subjects n has been tested.
The different light levels, room temperatures, etc. will have an equally likely "chance" of being assigned to each treatment level.
Thus there is no systematic bias leading to running of one condition at the same time of day or only in warm rooms or only when the lights were bright.
- Enumerate the factor/level combinations and the number of trials:
Experiment Factor Level Trial Number Number ======================================== 1 low 1 2 low 2 3 low 3 4 high 1 5 high 2 6 high 3 ========================================- Next a random number generator to select six numbers from the set {1,2,3,4,5,6} without replacement, using a uniform distribution.
Example: 4,1,5,2,3,6
- Run experiment 3, then 1, then 5, ..., then 6.
- Suppose a secondary factor changed after experiments 4,1,5,2.
Then experiments 3 and 6 are biased. But the bias will equally affect your measurements of the "low" (experiment 3) and "high" factors (experiment 6)!