Langston, Research Methods, Notes 14 -- Non-experimental designs

I.  Goals.
A.  Kinds of single-N (small-N) designs.
B.  Methods.
C.  Quasi experiments.
D.  Pros and cons.

II.  Kinds of single-N (small-N) designs.  These are technically experiments with only one participant, but we'll be a little more general and include experiments where you have so few participants that you only have one group (no control group).
A.  Case studies:  A source of rich information about a single individual.  Find out everything you can about one case, try to find some general principles that could be applied to other people with similar problems (Freud, Skinner).
B.  Single-N experiment:  Do an experiment with just one participant.
C.  Small group:  An experiment like B, but with a group of participants, no control group.

Top

III.  Methods.
A.  Baseline designs:
1.  Simple:  Measure, treat, measure (AB design).  Schematically:

First, you take a baseline measure.  Then you treat.  Then you look for changes from baseline.  If performance goes up (or down) after treatment, you want to conclude that the change was caused by the treatment.
You can see the inherent weakness here pretty quickly.  The difference could be due to chance variation (performance might make lots of big changes from time to time, and you just picked up one of these regular fluctuations), or some other event (a confound) might have occurred that really made the change.  This is bad.
2.  Complex:  Measure, treat, measure, withdraw treatment, measure... (a.k.a. ABAB or reversal design).  Schematically:

First you get a baseline.  Then, you treat and look for a change.  Then you remove treatment and look for a return to baseline.  Then you treat again.  The idea is that if performance goes up when you add treatment and down when you remove it, the treatment is causing those changes.
Problems:
a.  Ethics:  Can you remove a treatment that works from a patient who needs it?
b.  You can only do this with something that will return to baseline when you remove treatment.  So, if the treatment has a lasting effect, this won't work.
B.  Multiple baseline:  Introduce treatment at different times for different aspects of a situation and look for change only after treatment, regardless of when it's introduced.  Schematically:

1.  Individuals:  Each graph (A, B, C) represents a different individual (who are ideally all in the same situation, like the same mental institution).  You introduce treatment for A earlier than B, and B earlier than C.  If each person only improves after treatment, it's probably not a confound causing the effect (if it was a confound, then B and C would have improved at the same time A improved).
2.  Behaviors:  A, B, and C could also be three behaviors in one person.  For example, a person might have obsessive thoughts about the stove being on, the baby falling in the toilet, and a counting ritual.  You could introduce therapy for each obsession at a different time to demonstrate that the therapy is effective for solving the person's problem.
3.  Situations:  If a person is overly aggressive with family, friends, and coworkers, then A, B, and C are three situations.  You could introduce therapy for aggression in each situation, and show that it's the therapy that improves relations.

Top

IV.  Quasi experiments.  Something about the design can compromise internal validity and makes causal conclusions difficult.  What is that something?  No random assignment.
A.  Ex post facto (after the fact).  You look at the effect of some intervention or event on people.  For example, you could look at the effect of a hurricane on people's mood.  The design can look a lot like an experiment:  Take people who were in the hurricane and some who weren't, compare mood.  You can even match these people on some variables to make sure they're similar.  But, since you didn't randomly assign and manipulate the presence of the hurricane, you can't tell for sure that differences were caused by the “IV.”
B.  One-shot case study:  You get the case after the “treatment” has occurred.  So, you can't get a baseline to compare to.  For example, you get an alcoholic with memory problems.  You can select a comparison person (same age, profession, social class, etc.) and compare the two, but without random assignment, it's all suspect.
C.  Interrupted-time-series.  An extended AB design where you didn't introduce the treatment.  For example, I might hypothesize that heat increases violence.  I could measure violence in the winter, and again in the summer.  If I go for several years, it could be an ABAB design.
D.  Participant variables.  You want to “manipulate” something like sex of the participant.  All of the correlation confounds apply.  You can try to match, but it's hard, and probably not entirely possible.

Top

V.  Pros and cons.
A.  Pros:
1.  Source of ideas:  Use these to suggest experiments to perform.
2.  Therapeutic innovation:  If you get an interesting case, study it carefully, try various therapies, see if you can come up with some general principles to apply to other individuals.
3.  Study rare phenomena:  If you only get one neuropsychological case of a patient who loses the ability to name fruits after a head injury, then that's all you have to study.
4.  Challenge assumptions:  If a theory says “If p then q” (like “If you have bulimia, then you've repressed memories of childhood sexual abuse”), all you need is one case of a person with bulimia who isn't repressing memories of sexual abuse to prove the hypothesis wrong.
B.  Cons:
1.  Cause-effect relationships are hard to determine.  With no random assignment and no control group, you're never really sure if changes are due to treatment or a confound.
2.  Biases:  Confirmation biases can influence data collection.  Plus, the experimenter is usually a participant in the research situation (like the therapist), which introduces numerous other sources of bias.
3.  Generalizability:  Can you really make conclusions about a population on the basis of a single case?
Conclusion:  There are problems with these designs, but in situations where it's your only option, they are better than nothing.

Top

Research Methods Notes 14
Will Langston