Research Methods
for the digitally inclined
Stephen R. Schmidt


Experimental Control

How to avoid threats to validity

I.  Introduction:  Goals

II.  Types of Experimental Control

III.  Means of Achieving Control

IV. Summary

I.  Introduction:  Goals

A.  Infer true causal relations

 1.  Rule out alternative explanations
  (threats to validity)
 2.  Hold žall other things equalÓ
  constant influence of extraneous factors

B.  Determine true functional relations

 1.  control over the levels of a variable

 2.  Hold all else constant

Example:   Memory as a function of  retention interval

Jenkins & Dallenback (1924)

Two subjects studied non-sense syllables
They were tested either immediately, 1, 2, 4 or 8 hours later
During the retention interval they either were awake doing daily activities, or they slept.


Classic Shape of the curve: (Woodworth, 1938)


Rubin, Hinton, & Wenzel (1999)

    Used a continuous recall task to get a lot of data to precisely model the time course of retention.

    The best fitting curve was the sum of three different log functions.


II.  Types of Experimental Control

A.  Experimental versus Control groups

Experimental group:  treatment
Control group: no treatment
        Control group is necessary for a comparison

Example:  use of a computer to aid instruction

B.  Control over the levels or values of the independent variable

Example:  the effects of cigarette smoking on the number of colds a person gets in a year
 - must control cigarette smoking.

C.  Control over the experimental setting

 - preventing extraneous variables

Example:  controlling the noise in an auditory perception experiment

D.  Control Variable:

A potential independent variable that is held constant across conditions in an experiment.

Example:  survey of attitudes toward sexual harassment in the workplace

Control:  gender, age, race, etc.

III.  Means of Achieving Control

A. Randomization

1.  Random selection

   Each person in a population has an equal opportunity to participate in the research.

- issue external validity of the results
- not really possible

2.  Random Assignment:

Each participant or item has an equal opportunity to be in each treatment condition.

- easy to accomplish
- necessary to protect internal validity

A.  Randomization (cont)

3.  What randomization accomplishes

 a) avoids systematic differences

 b) controls both known and unknown variables.

4.  Things to randomize:

5.  How to randomize

Random numbers table (p. 446 )

10480 15011 01536 02011
22368 46573 25595 85393
24130 48360 22527 97265
42167 93093 06243 61658
37570 39975 81837 16656
77921 06907 11008 42751
99562 72095 56420 69994
96301 91977 05463 07972

B.  Counterbalancing

Definition:  A means to systematically vary important control variables so that their influences are equally distributed across conditions.

Insures that each condition precedes and follows each other condition an equal number of times.
Counterbalancing (cont)

1) Things to counterbalance:

 - order of treatments
 - order of questions
 - position of items
 - subject variables (e.g., reading level)

With 2 levels of an independent variable:

2) Intra-subject counterbalancing:

Example:  effects of caffeine on reading speed
Without counterbalancing:  Caffeine, No Caffeine

With counterbalancing:
No Caffeine, Caffeine, Caffeine, No Caffeine

3) Intragroup Counterbalancing:

Group 1:  A  B
Group 2:  B  A

Example        Group 1: Caffeine, No Caffeine
                     Group 2:  No Caffeine, Caffeine

With four levels of an independent variable:

 A= caffeine
 B= cold medicine
 C= alcohol
 D= no medication

Latin Square

4)  Counter balancing a subject variable:

(e.g. reading speed)

Rank the subjects:  1, 2, 3, 4, 5, 6, . . .
Cond 1:  1, 4, 5, 8
Cond 2:  2, 3, 6 ,7

C.  Matching

 Definition:  A means of controlling a variable by attempting to equate it across conditions.

Things to match:

Matching procedures:

1) Hold the variable constant:

 example:  hold reading level constant in drug study.

 problem:  limits external validity

2.  Matched Groups:

 hold the average across groups constant

Example:  construct 2 groups that have the same mean reading level

Problem: may hold the mean constant, but the range and variance may be very different.

3.  Incorporate the Variable as an additional independent variable.

Caffeine No Caffeine
Reading Level 1    
Reading Level 2    
Reading Level 3    

Problem:  may lead to very large experimental designs.

4.  Yoking

   Pair subjects or items on an important dimension, and then randomly assign one member of each pair to each condition.

Brady (1958) "executive monkey"
(an example in yoking)

Results:  All four executive monkeys developed ulcers, while none of the yoked monkeys did.
Conclusion:  The stress of making decisions caused the ulcers.

And now for the rest of the story!

Executive Monkeys Part Deux

IV. Summary

A.  Goal: to conduct research that is free from threats to validity.

B.  This means exercising control over the important variables

C.  Primary Means of Achieving that control:

About these pages | Syllabus & Lecture Notes | Projects | Sample Tests | Related Links | Exit