Chapter 11: Quasi-Experimental Designs

Although the randomized trial is considered the optimal design for testing cause-and-effect hypotheses, the necessary restrictions of a randomized trial are not always possible within the clinical environment.¹ Depending on the nature of the treatment under study and the population of interest, use of randomization and control groups may not be feasible. The specification of inclusion and exclusion criteria will often reduce generalizability and limit the range of patients that can be included.

Quasi-experimental designs utilize similar structures to experimental designs, but will lack either random assignment or comparison groups, or both. They often involve nonequivalent groups that may differ from each other in many ways in addition to differences between treatment conditions.² Therefore, the degree of control is reduced. Many studies incorporate quasi-experimental elements because of the limitations of clinical conditions. These designs present reasonable alternatives to the randomized trial, as long as the researcher carefully documents subject characteristics, controls the research protocol, and uses blinding as much as possible. The conclusions drawn from these studies must take into account the potential biases of the sample, but may provide important information, nonetheless.³

One-Group Pretest-Posttest Design

The one-group pretest-posttest design is a quasi-experimental design that involves one set of repeated measurements taken before and after treatment on one group of subjects (see Figure 11.1). The effect of treatment is determined by measuring the difference between pretest and posttest scores. In this design, the independent variable is time, with two levels (pretest and posttest). Treatment is not an independent variable because all subjects receive the intervention.

In this study, the researchers saw significant improvements in outcome variables, and concluded that the treatment was successful. We must hold conclusions drawn from this design as suspect, however. The design is weak because it has no comparison group, making it especially vulnerable to threats to internal validity. Although the researcher can demonstrate change in the dependent variable by comparing pretest and posttest scores, there is always the possibility that some events other than the experimental treatment occurred within the time frame of the study that caused the observed change. Therefore, this design is particularly threatened by history and maturation effects. In addition, the influence of testing, instrumentation, statistical regression, and selection interaction effects cannot be ruled out. External validity is also limited by potential interactions with selection because there is no comparison group.

The one-group pretest-posttest design may be defended, however, in cases where previous research has documented the behavior of a control group in similar circumstances. For instance, other studies may have shown that shoulder pain does not improve in this population over a 4-week period without intervention. On that basis, we could justify using a single experimental group to investigate just how much change can be expected with treatment. This documentation might also allow us to defend the lack of a control group on ethical grounds. The design is also reasonable when the experimental situation is sufficiently isolated so that extraneous environmental variables are effectively controlled, or where the time interval between measurements is short so that temporal effects are minimized.² For instance, in studies where data collection is completed within a single testing session, temporal threats to internal validity will be minimal, although testing effects remain uncontrolled. Under all circumstances, however, this design is not considered a true experiment, and should be expanded whenever possible to compare two groups.

Analysis of One-Group Pretest-Posttest Designs. A t-test for paired comparisons is usually used to compare pretest and posttest mean scores. With ordinal data or small samples, the sign test or the Wilcoxon signed-ranks test can be used.

One-Way Repeated Measures Design Over Time

Many research questions that deal with the effects of treatment on physiological or psychological variables are concerned with how those effects are manifested over time. As an extension of the pretest-posttest design, the repeated measures design is naturally suited to assessing such trends.⁵ Multiple measurements of the dependent variable are taken within prescribed time intervals. The intervention may be applied once, or it may be repeated in between measurements (see Figure 11.2).

Once again, in this example time is the independent variable. Every subject is evaluated at each time interval, making it a repeated measure. The design is quasi-experimental because there is no randomization of the order of treatment and no comparison group. Without a control group internal validity is threatened, as it is not possible to discern if changes would have occurred over time without the intervention.

This design may be appropriate, however, when the time course of a disease or condition is predictable. For example, several studies have shown that strength declines in a linear fashion in patients with amyotrophic lateral sclerosis (ALS).^7,8 Therefore, studies using this population for trials of drugs or other interventions may reasonably follow patients over time without needing a control or placebo group.⁹

Analysis of One-Way Repeated Measures Designs Over Time. When time is the independent variable, a one-way repeated measures analysis of variance can be performed. Analysis may also include polynomial contrasts to describe trends over time.

Time Series Design

Time-series designs are based on the application of multiple measurements, before and after treatment, to document patterns or trends of behavior (see Figure 11.3). These designs are often used to study community interventions¹⁰ and policy changes.¹¹ They have also been adapted by behavioral analysts for the study of single subjects' responses over time (see Chapter 12). Basic time series designs have been defined by Cook and Campbell.²

This is an example of an interrupted time-series design (see Figure 11.4), so named because it involves a series of measurements over time that are "interrupted" by one or more treatment occasions. It is considered a quasi-experimental design because only one group is studied. The independent variable is time; that is, each measurement interval represents one level of time.^* The research question concerns trends across these time intervals. The number of observations can vary, depending on the stability of the dependent variable. In some studies, researchers may extend pretest or posttest periods if the data are very variable, in an effort to stabilize responses before initiating treatment or ending the observations.

This design may be considered an extension of the one-group pretest-posttest design. It offers more control, however, because the multiple pretests and posttests act as a pseudocontrol condition, demonstrating maturational trends that naturally occur in the data or the confounding effects of extraneous variables. It is most effective when serial data can be collected at evenly distributed intervals, avoiding confounding by extraneous temporal factors.¹² To illustrate, consider several possible outcomes for an interrupted time-series design, shown in Figure 11.5. A series of observations are taken at times 1 through 8 (O₁–O₈), with the introduction of treatment at point X. In all three patterns, a similar increase in the dependent variable is seen from O₄ to O₅. In Pattern A we would be justified in assuming that treatment has an effect, as no change occurred prior to intervention. In Pattern B, however, it would be misleading to make this interpretation, as the responses are continually increasing within the baseline measures. Although Pattern C also shows an increase in the dependent variable from O₄ to O₅, which would look like a treatment effect if these were the only two measurements taken, we can see from the erratic pattern changes before and after treatment that this conclusion is unwarranted.

The greatest threat to internal validity in the time series design is history. There is no control over the possible coincidental occurrence of some extraneous event at the same time that treatment is initiated. The level and trend of the pre-intervention data, however, do present a form of control for the post-intervention segment¹¹ (see Chapter 12 for further discussion of level and trend). This is illustrated in the data in Figure 11.4.

Most other threats to internal validity are fairly well controlled by the presence of multiple measurements; that is, their effects are not eliminated, but we can account for them. For instance, if instrumentation or testing effects are present, we should see changes across the pretest scores. External validity of a time-series design is limited to situations where repeated testing takes place.

Several variations can be applied to this design. Comparisons may include two or more groups. Treatment may be administered one time only with follow-up measurements, or intervention may be continued throughout the posttest period. In a third variation, treatment may be started after a series of pretest measurements, and then withdrawn after a specified time period, with measurements continuing into the withdrawal period. The withdrawal design does help to account for history effects. If the behavior improves with treatment, and reverts back to baseline levels when treatment is withdrawn, a strong case can be made that extraneous factors were not operating. In a fourth model, called a multiple baseline design, each subject is tested under baseline conditions, and the introduction of treatment is staggered, again controlling for history and maturation effects. These models have been incorporated into single-subject designs (see Chapter 12).

Analysis of Time Series Designs. Many researchers use graphic visual analysis as the primary means of interpreting time-series data. There is considerable disagreement as to the validity of visual analysis. Statistical techniques for time-series analysis involve multivariate methods. A model called the autoregressive integrated moving average (ARIMA) is often used to accommodate for serial scores and weighs heavily on the observations that fall closer to the point at which treatment is introduced (see Chapter 12 for a discussion of serial dependency and autocorrelation).¹³ It analyzes the trends in the data and provides an estimate of the variability to be expected among future observations.¹⁴ To be effective, the ARIMA procedure requires at least 25 data points in each phase.¹⁵ For a more detailed description of this procedure, consult Cook and Campbell.² A technique called segmented regression analysis (shown in Figure 11.4) has also been used to look at patterns of change over time in the level and slope of regression lines for different segments of the data.¹⁶

Nonequivalent Pretest-Posttest Control Group Design

There are many research situations in the social, clinical, and behavioral sciences where groups are found intact or where subjects are self-selected. The former case is common in a clinic or school where patients or students belong to fixed groups or classes. The latter case will apply when attribute variables are studied or when volunteers are recruited. The nonequivalent pretest-posttest control group design (see Figure 11.6) is similar to the pretest-posttest experimental design, except that subjects are not assigned to groups randomly. This design can be structured with one treatment group and one control group or with multiple treatment and control groups.

In the study of therapy for neck pain, the patients are members of intact groups by virtue of their diagnosis. In the chair exercise study, subjects self-selected their group membership.

Although the nonequivalent pretest-posttest control group design is limited by the lack of randomization, it still has several strengths. Because it includes a pretest and a control group, there is some control over history, testing and instrumentation effects. The pretest scores can be used to test the assumption of initial equivalence on the dependent variable, based on average scores and measures of variability. The major threat to internal validity is the interaction of selection with history and maturation. For instance, if those who chose to participate in chair exercises were stronger or more motivated patients, changes in outcomes may have been related to physiological or psychological characteristics of subjects. These characteristics could affect general activity level or rate of healing. Such interactions might be mistaken for the effect of the exercise program. These types of interactions can occur even when the groups are identical on pretest scores.

Analysis of Nonequivalent Pretest-Posttest Designs. Several statistical methods are suggested for use with nonequivalent groups, including the unpaired t-test (with two groups), analysis of variance, analysis of covariance, analysis of variance with matching, and analysis of variance with gain scores. Ordinal data can be analyzed using the Mann-Whitney U test. Nonparametric tests may be more appropriate with nonequivalent groups, as variances are likely to be unequal. Preference for one approach will depend in large part on how groups were formed and what steps the researcher can take to ensure or document initial equivalence. Tests such as the t-test, analysis of variance and chi square are often used to test for differences in baseline measures. Regression analysis or discriminant analysis may be the most applicable approach to determine how the dependent variable differentiates the treatment groups, while adjusting for other variables. Statistical strategies must include mechanisms for controlling for group differences on potentially confounding variables.

Historical Controls

Another strategy for comparing treatments involves the use of historical controls who received a different treatment during an earlier time period.

As this example illustrates, a nonconcurrent control group may best serve the purpose of comparison when ethical concerns may preclude a true control group. When the researcher truly believes that the experimental intervention is more effective than standard care, the use of historical controls provides a reasonable alternative.²⁰ This approach has been used in cancer trials, for example, when protocols in one trial act as a control for subsequent studies.²¹ The major advantage of this approach is its efficiency. Because all subjects are assigned to the experimental condition, the total sample will be smaller and the results can be obtained in a shorter period of time.

The disadvantages of using historical controls must be considered carefully, however. Studies that have compared outcomes based on historical controls versus randomly allocated controls have found positive treatment effects with historical controls that randomized trials have not been able to replicate.^22,23 The most obvious problem, therefore, is the potential for confounding because of imbalances in characteristics of the experimental and historical control groups. For this approach to work, then, the researcher must be diligent in establishing a logical basis for group comparisons. This means that the historical controls should not simply be any patients described in the literature, or those treated at another time or another clinic.^20,24 It is reasonable, however, as in the kidney transplant example, to consider using groups that were treated within the same environment, under similar conditions, where records of protocols were kept and demographics of subjects can be obtained. This approach may prove useful as large clinical data bases are accumulated within a given treatment setting.

Analysis of Designs with Historical Controls. Researchers often use the independent samples t-test to compare current subjects with historical subjects, although there is an inherent flaw in this approach because there is no assumption of equivalence between the groups. The Mann-Whitney U test may be used with ordinal data. Chi square will allow the researcher to determine if there is an association among categorical variables. Multiple regression, logistic regression or discriminant analysis can be done, using group membership as a variable, to analyze differences between the groups while accounting for other variables.

Nonequivalent Posttest-Only Control Group Design

Nonequivalent designs are less interpretable when only posttest measures are available. The nonequivalent posttest-only control group design (see Figure 11.7), also called a static group comparison,²⁵ is a quasi-experimental design that can be expanded to include any number of treatment levels, with or without a control group. This design uses existing groups who have and have not received treatment.

To draw conclusions from this comparison, we would have to determine if variables other than the exercise program could be responsible for outcomes. Confounding factors should be identified and analyzed in relation to the dependent variable. For instance, in the cardiac rehabilitation example, researchers considered the subjects' age, years of education and occupational skill.²⁶ Although the static group comparison affords some measure of control in that there is a control group, internal validity is severely threatened by selection biases and attrition. This design is inherently weak because it provides no evidence of equivalence of groups before treatment. Therefore, it should be used only in an exploratory capacity, where it may serve to generate hypotheses for future testing. It is essentially useless in the search for causal relationships.

Analysis of Nonequivalent Posttest-Only Designs. Because this design does not allow interpretation of cause and effect, the most appropriate analysis is a regression approach, such as discriminant analysis. Essentially, this design allows the researcher to determine if there is a relationship between the presence of the group attribute and the measured response. An analysis of covariance may be used to account for the effect of confounding variables.