Single-Factor Designs for Independent Groups
A single-factor design, also called a one-way design, is used to structure the investigation of one independent variable. The study may include one or more dependent variables.
Pretest-Posttest Control Group Design
The pretest-posttest control group design is the basic structure of a randomized controlled trial. It is used to compare two or more groups that are formed by random assignment. One group receives the experimental variable and the other acts as a control. These independent groups are also called treatment arms of the study. Both groups are tested prior to and following treatment. The groups differ solely on the basis of what occurs between measurements. Therefore, changes from pretest to posttest that appear in the experimental group but not the control group can be reasonably attributed to the intervention. This design is considered the scientific standard in clinical research for establishing a cause-and-effect relationship.
The pretest-posttest control group design can be configured in several ways. Figure 10.1 illustrates the simplest configuration, with one experimental group and one control group.
Pretest-posttest control group design; the basic structure of a randomized controlled trial (RCT).
Example of a Pretest-Posttest Control Group Design
Researchers conducted a randomized controlled trial to study the effect of a supervised exercise program for improving venous hemodynamics in patients with chronic venous insufficiency.6 They randomly assigned 31 patients to two groups. The experimental group received physical therapy with specific exercises for calf strengthening and joint mobility. The control group received no exercise intervention. Both groups received compression hosiery. Dynamic strength, calf pump function and quality of life were assessed at baseline and after 6 months of exercise.
Measurements for the control group are taken within intervals that match those of the experimental group. The independent variable has two levels, in this case exercise intervention and control. The absence of an experimental intervention in the control group is considered a level of the independent variable. As this example illustrates, a study may have several dependent variables that are measured at pretest and posttest.
The pretest-posttest design can also be used when the comparison group receives a second form of the intervention. The two-group pretest-posttest design (see Figure 10.2) incorporates two experimental groups formed by random assignment.
Two-group pretest-posttest design.
Example of a Two-Group Pretest-Posttest Design
Researchers conducted a randomized controlled trial to study the effect of semantic treatment on verbal communication in patients who experienced aphasia following a stroke.7 They randomly assigned 58 patients to two groups. Speech therapists provided semantic treatment to the experimental group. The control group received speech therapy focused on word sounds. Verbal communication was assessed using the Amsterdam Nijmegen Everyday Language Test. Both groups were assessed at the start of the study and following 7 months of treatment.
Researchers use this approach when a control condition is not feasible or ethical, often comparing a "new" treatment with an "old" standard or alternative treatment. Even though there is no traditional control group, this design provides experimental control because we can establish initial equivalence between groups formed by random assignment. In this example, the word sound group acts as a control for the semantic treatment group and vice versa. If one group improves more than the other, we can attribute that difference to the fact that one treatment was more effective. This design is appropriate when the research question specifically addresses interest in a difference between two treatments, but it does not allow the researcher to show that treatment works better than no intervention.
The multigroup pretest-posttest control group design (see Figure 10.3) allows researchers to compare several treatment and control conditions.
Multigroup pretest-posttest design.
Example of a Multigroup Pretest-Posttest Design
Researchers wanted to determine the effectiveness of aquatic and on-land exercise programs on functional fitness and activities of daily living (ADLs) in older adults with arthritis.8 Participants were 30 volunteers, randomly assigned to aquatic exercise, on-land exercise or a control group. The control group was asked to refrain from any new physical activity for the duration of the study. Outcomes included fitness and strength measures, and functional assessments before and after an 8-week exercise program.
As these examples illustrate, the pretest-posttest control group design can be expanded to accommodate any number of levels of one independent variable, with or without a traditional control group. This design is strong in internal validity. Pretest scores provide a basis for establishing initial equivalence of groups, strengthening the evidence for causal factors. Selection bias is controlled because subjects are randomly assigned to groups. History, maturation, testing, and instrumentation effects should affect all groups equally in both the pretest and posttest. The only threat to internal validity that is not controlled by this design is attrition.
The primary threat to external validity in the pretest-posttest control group design is the potential interaction of treatment and testing. Because subjects are given a pretest, there may be reactive effects, which would not be present in situations where a pretest is not given.
Analysis of Pretest-Posttest Designs. Pretest-posttest designs are often analyzed using change scores, which represent the difference between the posttest and pretest.* With interval-ratio data, difference scores are usually compared using an unpaired t-test (with two groups or a one-way analysis of variance (with three or more groups). With ordinal data, the Mann-Whitney U-test can be used to compare two groups, and the Kruskal-Wallis analysis of variance by ranks is used to compare three or more groups. The analysis of covariance can be used to compare posttest scores, using the pretest score as the covariate. The design can be analyzed as a two-factor design, using a two-way analysis of variance with one repeated factor, with treatment as one independent variable and time (pretest and posttest) as the second (repeated) factor. Discriminant analysis can also be used to distinguish between groups with multiple outcome measures.
Posttest-Only Control Group Design
The posttest-only control group design (see Figure 10.4) is identical to the pretest-posttest control group design, with the obvious exception that a pretest is not administered to either group.
Posttest-only control group design.
Example of Posttest-Only Control Group Design
A study was designed to test the hypothesis that high-risk patients undergoing elective hip and knee arthroplasty would incur less total cost and shorter length of stay if inpatient rehabilitation began on postoperative day 3 rather than day 7.9 Eighty-six patients who were older than 70 years were randomly assigned to begin rehabilitation on day 3 or day 7. The main outcome measures were total length of stay and cost from orthopedic and rehabilitation admissions.
In this study of hospital cost and length of stay, the dependent variables can only be assessed following the treatment condition. This design is a true experimental design which, like the pretest-posttest design, can be expanded to include multiple levels of the independent variable, with a control, placebo or alternative treatment group.
Because this design involves random assignment and comparison groups, its internal validity is strong, even without a pretest; that is, we can assume groups are equivalent prior to treatment. Because there is no pretest score to document the results of randomization, this design is most successful when the number of subjects is large, so that the probability of truly balancing interpersonal characteristics is increased.
The posttest-only design can also be used when a pretest is either impractical or potentially reactive. For instance, to study the attitudes of health care personnel toward patients with AIDS, we might use a survey instrument that asked questions about attitudes and experience with this population. By using this instrument as a pretest, subjects might be sensitized in a way that would influence their scores on a subsequent posttest. The posttest-only design avoids this form of bias, increasing the external validity of the study.
Analysis of Posttest-Only Designs. With two groups, an unpaired t-test is used with interval-ratio data, and a Mann-Whitney U-test with ordinal data. With more than two groups, a one-way analysis of variance or the Kruskal-Wallis analysis of variance by ranks should be used to compare posttest scores. An analysis of covariance can be used when covariate data on relevant extraneous variables are available. Regression or discriminant analysis procedures can also be applied.
Multi-Factor Designs for Independent Groups
The designs presented thus far have involved the testing of one independent variable, with two or more levels. Although easy to develop, these single-factor designs tend to impose an artificial simplicity on most clinical and behavioral phenomena; that is, they do not account for simultaneous and often complex interactions of several variables within clinical situations. Interactions are generally important for developing a theoretical understanding of behavior and for establishing the construct validity of clinical variables. Interactions may reflect the combined influence of several treatments or the effect of several attribute variables on the success of a particular treatment.
A factorial design incorporates two or more independent variables, with independent groups of subjects randomly assigned to various combinations of levels of the two variables. Although such designs can theoretically be expanded to include any number of variables, clinical studies usually involve two or three at most. As the number of independent variables increases, so does the number of experimental groups, creating the need for larger and larger samples, which are typically impractical in clinical situations.
Factorial designs are described according to their dimensions or number of factors, so that a two-way or two-factor design has two independent variables, a three-way or three-factor design has three independent variables, and so on. These designs can also be described by the number of levels within each factor, so that a 3 × 3 design includes two variables, each with three levels, and a 2 × 3 × 4 design includes three variables, with two, three and four levels, respectively.
A factorial design is diagrammed using a matrix notation that indicates how groups are formed relative to levels of each independent variable. Uppercase letters, typically A, B and C, are used to label the independent variables and their levels. For instance, with two independent variables, A and B, we can designate three levels for the first one (A1, A2 and A3) and two levels for the second (B1, B2).
The number of groups is the product of the digits that define the design. For example, 3 × 3 = 9 groups; 2 × 3 × 4 = 24 groups. Each cell of the matrix represents a unique combination of levels. In this type of diagram there is no indication if measurements within a cell include pretest-posttest scores or posttest scores only. This detail is generally described in words.
Two-Way Factorial Design. A two-way factorial design (see Figure 10.5) incorporates two independent variables, A and B.
A. Two-way factorial design. B. Main effects for two-way factorial design.
Example of a Two-Way Factorial Design
Researchers were interested in studying the effect of intensity and location of exercise programs on the self-efficacy of sedentary women.10 Using a 2 × 2 factorial design, subjects were randomly assigned to one of four groups, receiving a combination of moderate or vigorous exercise at home or a community center. The change in their exercise behavior and their self-efficacy in maintaining their exercise program was monitored over 18 months.
In this example, the two independent variables are intensity of exercise (A) and location of exercise (B), each with two levels (2 × 2). One group (A1B1) will engage in moderate exercise at home. A second group (A2B1) will engage in vigorous exercise at home. The third group (A1B2) will engage in moderate exercise at a community center. And the fourth group (A2B2) will engage in vigorous exercise at a community center. The two independent variables are completely crossed in this design, which means that every level of one factor is represented at every level of the other factor. Each of the four groups represents a unique combination of the levels of these variables, as shown in the individual cells of the diagram in Figure 10.5A. For example, using random assignment with a sample of 60 patients, we would assign 15 subjects to each group.
This design allows us to ask three questions of the data: (1) Is there a differential effect of moderate versus vigorous exercise? (2) Is there a differential effect of exercising at home or a community center? (3) What is the interaction between intensity and location of exercise? The answers to the first two questions are obtained by examining the main effect of each independent variable, with scores collapsed across the second independent variable, as shown in Figure 10.5B. This means that we can look at the overall effect of intensity of exercise without taking into account any differential effect of location. Therefore, we would have 30 subjects representing each intensity. The main effect of location is also analyzed without differentiating intensity. Each main effect is essentially a single-factor experiment.
The third question addresses the interaction effect between the two independent variables. This question represents the essential difference between single-factor and multifactor experiments. Interaction occurs when the effect of one variable varies at different levels of the second variable. For example, we might find that moderate exercise intensity is more effective in changing exercise behavior, but only when performed at a community center.
This example illustrates the major advantage of the factorial approach, which is that it gives the researcher important information that could not be obtained with any one single-factor experiment. The ability to examine interactions greatly enhances the generalizability of results.
Three-Way Factorial Design. Factorial designs can be extended to include more than two independent variables. In a three-way factorial design (see Figure 10.6), the relationship among variables can be conceptualized in a three-dimensional format. We can also think of it as a two-way design crossed on a third factor.
Three-way factorial design.
For example, we could expand the exercise study shown in Figure 10.5 to include a third variable such as frequency of exercise. We would then evaluate the simultaneous effect of intensity, location and frequency of exercise. We could assign subjects to exercise 1 day or 3 days per week. Then we would have a 2 × 2 × 2 design, with subjects assigned to one of 8 independent groups (see Figure 10.6).
A three-way design allows several types of comparisons. First, we can examine the main effect for each of the three independent variables, collapsing data across the other two. We can examine the difference between the two intensities, regardless of the effect of location or frequency. We can test the difference between the two locations, regardless of intensity or frequency. And we can evaluate the effect of frequency of exercise, regardless of intensity or location. Each of the three main effects essentially represents a single-factor study for that variable.
Then we can examine three double interactions: intensity × location, intensity × frequency, and location × frequency. For example, the interaction between intensity and location is obtained by collapsing data across the two levels of frequency of exercise. Each double interaction represents a two-way design. Finally, we can examine the triple interaction of intensity, location and frequency. This interaction involves analyzing the differences among all 8 cells.
Many clinical questions have the potential for involving more than one independent variable, because response variables can be influenced by a multitude of factors. In this respect, the compelling advantage of multidimensional factorial designs is their closer approximation to the "real world." As more variables are added to the design, we can begin to understand responses, increasing construct validity of our arguments. The major disadvantages, however, are that the sample must be extremely large to create individual groups of sufficient size and that data analysis can become cumbersome.
Analysis of Factorial Designs. A two-way or three-way analysis of variance is most commonly used to examine the main effects and interaction effects of a factorial design.
When a researcher is concerned that an extraneous factor might influence differences between groups, one way to control for this effect is to build the variable into the design as an independent variable. The randomized block design (see Figure 10.7) is used when an attribute variable, or blocking variable, is crossed with an active independent variable; that is, homogeneous blocks of subjects are randomly assigned to levels of a manipulated treatment variable. In the following example, we have a 2 × 3 randomized block design, with a total of 6 groups.
Example of a Randomized Block Design
A study was performed to assess the action of an antiarrhythmic agent in healthy men and women after a single intravenous dose.11 Researchers wanted to determine if effects were related to dose and gender. Twenty-four subjects were recruited, 12 men and 12 women. Each gender group was randomly assigned to receive 0.5, 1.5 or 3.0 mg/kg of the drug for 2 minutes. Therefore, 4 men and 4 women received each dose. Through blood tests, volume of distribution of the drug at steady state was assessed before and 72 hours after drug administration. The change in values was analyzed across the six study groups.
In studying the drug's effect, the researchers were concerned that men and women would respond differently. We can account for this potential effect by using gender as an independent variable. We can then assume that responses will not be confounded by gender.
We can think of this randomized block design as two single-factor randomized experiments, with each block representing a different subpopulation. Subjects are grouped by blocks (gender), and then random assignment is made within each block to the treatment conditions. When the design is analyzed, we will be able to examine possible interaction effects between the treatment conditions and blocks. When this interaction is significant, we will know that the effects of treatment do not generalize across the block classifications, in this case across genders. If the interaction is not significant, we have achieved a certain degree of generalizability of the results.
For the randomized block design to be used effectively, the blocking factor must be related to the dependent variable; that is, it must be a factor that affects how subjects will respond to treatment. If the blocking factor is not related to the response, then using it as an independent variable provides no additional control to the design, and actually provides less control than had random assignment been used. Randomized block designs can involve more than two independent variables, with one or more blocking variables.
Generalization of results from a randomized block design will be limited by the definition of blocks. For example, classification variables, such as gender or diagnosis, are often used as blocking variables. The number of levels of these variables will be inherent. When the blocking factor is a quantitative variable, however, such as age, two important decisions must be made. First, the researcher must determine the range of ages to be used. Second, the number and distribution of blocks must be determined. Generally, it is best to use equally spaced levels with a relatively equal number of subjects at each level. If the researcher is interested in trends within a quantitative variable, three or more levels should be used to describe a pattern of change. For instance, if four age groups are delineated, we would have a clearer picture of the trends that occur with age than if only two levels were used.
Analysis of Randomized Block Designs. Data from a randomized block design can be analyzed using a two-way analysis of variance, multiple regression or discriminant analysis.
To this point, we have described multifactor designs in terms of two or three independent variables that are completely crossed; that is, all levels of variable A have occurred within all levels of variable B. This approach does not fit all multifactor analyses, however, when attribute variables are involved. Sometimes attribute variables cannot be crossed with all levels of other variables. Consider the following example.
Example of a Nested Design
An occupational therapist was interested in studying an intervention to facilitate motivational behaviors in individuals with psychiatric illness who had motivational deficits.12 The intervention was based on strategies of autonomy support. Patients were randomly assigned to either an experimental or control group, and to one of two different groups of therapists who carried out the treatments.
To study the effectiveness of the intervention, scores would be compared across 10 "therapists," each providing either the experimental treatment or control condition. If we used a traditional two-way design (10 × 2), all 10 levels of therapists would be crossed with both levels of treatment. This would allow the researcher to look at the main effect of therapists, to determine if differences were due to their application of the treatment. If a significant interaction occurred between therapist and treatment, it would mean that the effectiveness of intervention was dependent on which therapist provided it.
If we wanted to follow up on this interaction, we might suspect that less experienced therapists provided a different quality of intervention than more experienced therapists. To test this, we could divide our sample of therapists into two groups based on their years of experience: "less experienced" and "more experienced." This introduces a third independent variable, experience, with two levels. But these two levels cannot be crossed with the 10 levels of therapists; that is, the same therapist cannot appear in both experience groups. Therefore, "therapists" are nested within "experience." All levels of therapist and experience can be crossed with the two methods in this nested design (see Figure 10.8). Although this resembles a three-way randomized block design, it must be analyzed differently because the interactions of therapist × experience and therapist × experience × method cannot be assessed.
Most variables in clinical studies can be completely crossed; however, with certain combinations of attribute variables, a nested arrangement is required. Nesting is commonly used in educational studies where classes are nested in schools or schools are nested in cities. For instance, Edmundson and associates studied an educational program to reduce risk factors for cardiovascular disease.13 They evaluated the effect of the program on 6,000 students from 96 schools in four states. The schools were nested in states. Within each state the schools were randomly assigned to receive the program or a control condition.
Analysis of Nested Designs. An analysis of variance is used to test for main effects and relevant interactions. The dimensions of that analysis depend on how many variables are involved in the study. Nested designs require a complicated approach to analysis of variance, which goes beyond the scope of this book. See Keppel for discussion of analysis of nested designs.14 (pp. 550-565)