++
Clinicians and researchers will evaluate literature from three perspectives. First, the assessment will help to determine if the study has validity in terms of design and analysis. Second, the results of the study are examined to determine the importance of the effect of intervention, the accuracy of diagnostic tests, or the risks associated with prognostic variables. Finally, the clinician may consider how effectively the results can be applied to a particular patient's management.
++
Many useful references are available to guide the process of critical appraisal of published papers.2,3,4,5,6,7,8,9 Several of these guidelines are also used to assess methodological quality in systematic reviews (see Chapter 16).
+++
What Is the Study About?
++
Before we can assess the validity of a study, we must first understand its intent. Researchers begin this process at the top, by reading the title and abstract. Titles should be informative, but are often so abbreviated that the reader is unable to learn much about content other than the general topic. If the reader is interested in the topic, then the next step is to read the abstract. Abstracts of research papers will include fairly specific information about the purpose, subjects, method, results, and major conclusions of the presented work. When readers decide that the results and conclusions stated in the abstract could be applied to their practice, they should read the body of the paper.
++
The introduction to an article tells us about the purpose of the study and how the authors developed the research question. It informs readers by providing a review of literature that frames the theoretical foundation for the study. The following questions help to focus the content of the introduction:
++
What is the problem being investigated? Has it been clearly stated? Why is it important?
How has the author used the literature to form a sound and logical rationale?
What is the theoretical context for the study?
Are references appropriate and comprehensive?
What type of research does the study represent?
What is the specific purpose of the study?
What are hypotheses or guiding questions that form the basis of the study?
++
In the opening sentences of an article, the author should establish the problem being investigated. The background material should demonstrate that the researchers have thoughtfully and thoroughly synthesized the literature and related theoretical models. This synthesis should provide a rationale for pursuing this line of research. Readers should be convinced that the study was needed.
++
By the end of the introduction, the specific purpose or aims of the study should have been clearly stated and the research question should be evident. The authors of an intervention study should have explicitly stated the hypotheses that were to be tested. If not, however, readers should be able to determine what the researchers expected to find. For a diagnostic accuracy study, it should be clear if the intent is to compare two tests, to estimate the accuracy of one test, or to examine a test's utility across participant groups. Studies of prognosis should document the relevance of predictive variables and validity of outcomes.
+++
Are the Results of the Study Valid?
++
The foundation of a research report is the method section, providing the details of the study. This is where we learn enough to decide if the results and conclusions of the study are meaningful. Flaws or omissions detected in the methods section affect the usefulness of interpretations derived from the study. Table 34.1 shows the questions that should be asked to assess the validity of an intervention study. Questions relevant to diagnostic accuracy and prognosis are shown in Tables 34.2 and 34.3.
++
++
++
++
Readers must know who the subjects were in order to interpret the validity of a study's conclusions and to understand the extent to which the findings can be generalized or applied to specific patients. We can ask questions to determine whether sampling bias might have existed and to identify potentially confounding factors. Age, gender, diagnosis, comorbidities and level of function of the subjects are among the many factors that may affect the validity of findings. Inclusion and exclusion criteria should be specified so that the target population for the study is clearly identified.
++
The number of subjects in a study is a consideration when interpreting the results of the statistical analysis. Specifically, failure to demonstrate statistically significant effects should not be assumed to mean that no effect truly exists. The authors should present the results of a power analysis to determine the possibility of a Type II error resulting from an inadequate sample size.
++
For intervention studies, the description of subjects should include information about the accessible population and whether a probability or nonprobability sampling process was used. This information provides a foundation for deciding how well the findings can be generalized.
++
In diagnostic studies, the sample should reflect an appropriate spectrum of patients for whom the test would be applied. A diagnostic test is useful only to the extent it can identify those with varied degrees of the target disorder.10
++
For prognostic studies, authors should provide a complete description of the duration of the disease or disorder at the time patients entered the study.11 Patients should be at a similar stage of disease to demonstrate consistency of outcomes.
++
Based on the stated purpose of the project, readers should be able to judge the appropriateness of the design choices that were made. For studies of interventions, randomized designs are important for the control of internal validity, although quasi-experimental studies may still provide useful findings with sufficient description and controls. In diagnostic and prognostic studies, cohort designs are most useful for following subjects over time. Levels of evidence that are appropriate for these studies have been summarized in Chapter 16 (Table 16.1). It is important, however, to take these recommendations as guidelines only, understanding that the quality of the study must be examined to determine its applicability as evidence for clinical practice.
++
Information on study design will allow the reader to determine if the study is free from bias. As much as possible, researchers should incorporate blinding of subjects, testers, and investigators to protect against such bias. For intervention studies with two or more comparison groups, blinding provides some assurance that responses are not distorted because of inconsistent application of procedures or measurements. For diagnostic studies, those who administer and score tests should be blinded to the subject's true diagnosis. In prognostic studies, those who measure outcomes should be blinded to the subject's prior status.
++
The design will also provide a framework for understanding points of data collection, and the extent to which subjects are followed over time. For intervention studies, this has implications for understanding differences in trends and changes across groups. For prognosis studies, where the presence of a prognostic factor can precede outcomes by long periods, follow-up is most important to be sure that the study duration is long enough to detect the outcomes of interest.
++
The balance between efficacy and effectiveness should also be considered. When true experimental designs are employed, with the strict control inherent in these designs, readers should consider how to relate the findings to their "real world." In the "experimental" setting, where confounding factors and internal validity are rigorously controlled, the effects of manipulating or imposing the independent variable can be accepted with a high degree of confidence. But have these controls created such an "artificial" situation that it would be unrealistic to expect the same outcome when methods or procedures are implemented in clinical practice?
++
On the other hand, if quasi-experimental designs or descriptive methods are employed, readers must be alert to the possibility that extraneous variables have interfered with the interpretation of the results. The validity of an experimental or quasi- experimental study will depend in large part on how subjects are assigned to groups and whether or not a control group was included in the design. For example, when intact groups are used as in case-control designs, the author should specify the basis on which the subjects were selected.
++
Readers should know what specific instruments or tools were used for measurements. If these are standard and commercially available, the model numbers and names and addresses of manufacturers should be included. If standardized questionnaires or survey instruments are used, references must be cited. When appropriate, reliability of measurements should be documented. Such documentation may be referenced from previous studies, or the researcher may establish reliability within the context of the study being reported.
++
The description of procedures should report the sequence of events from beginning to end. Description of data collection methods should include who performed the measurements, what the subjects were asked to do, what the data collectors did, and when and how often measurements were taken. All of these activities must be described in sufficient detail so that readers could, in a similar setting with similar subjects, replicate the study procedures. Studies of interventions typically involve comparison of groups or conditions, and the author should document the extent to which these groups were treated equally, except for the experimental treatment.
++
Both descriptive and inferential statistical analyses should be identified. If inferential statistics were used, the acceptable significance level, such as α = .05, should have been established by the researchers and reported. The statistical tests should be described not merely by name, but by specifying to which data the analyses were applied. This information is necessary for readers to evaluate the appropriateness of the analyses. The use or misuse of statistics should be judged on the basis of two major factors: the nature of measurements (scale, reliability, linearity, and so on) and the study design (the number of groups or variables and frequency of measurement). Discrepancies in the proper application of statistical tests interfere with the statistical conclusion validity of the study and detract from the interpretation of the data.
++
Authors should report the degree of follow-up that was achieved and if attrition was present in the study sample. Reasons for attrition should be provided, to help determine if bias was present. In intervention studies, the author should specify if an intention to treat analysis was done, and how missing data were handled. The degree of follow-up is especially important for prognostic studies, to assure that outcomes are representative of the sample.
+++
Are the Results Meaningful?
++
The results section of a research report should contain the findings of the study without interpretation or commentary. Readers should find a narrative description of results, typically including test statistic values, such as t or F, odds ratios, or likelihood ratios.
++
It is rarely sufficient to simply determine that values are "significant," however. The usefulness of a study's results lies in the size of the effect that was demonstrated. Effect size tells us about the strength of the observed relationships. As we strive for greater application of evidence in clinical practice, we must understand just how much change we should expect from an intervention, how accurate a diagnostic will be, or how likely prognostic outcomes will occur. Statistics such as risk ratios, likelihood ratios, and number needed to treat will help us understand these effects. The precision of these estimates should be reflected in confidence intervals.
++
The narrative description of findings may be illustrated with or complemented by figures and tables to clarify and summarize the characteristics of the data. Readers should study graphs and tables carefully. The effect size indices are often presented in tabular fashion, rather than in the text. It is a useful exercise to compare data from tables with information found in the text, to be sure that the discussion of results follows what was actually reported.
+++
What Does It All Mean?
++
A research report ends with the authors' discussion and conclusions, putting the results of the study in context. The discussion section of an article should address each of the research questions, and should show how relevant literature helps to support the author's conclusions. Some useful questions to focus this part of the review include:
++
How does the author interpret results?
Did the author clarify if hypotheses were rejected or accepted?
What alternative explanations does the author consider for the obtained findings?
How are the findings related to prior reports?
What limitations are described? Are there limitations that are not addressed?
If results are not significant, does the author consider the possibility of Type II error?
Regardless of the statistical outcome, are the results clinically important?
Does the author discuss how the results apply to practice?
Does the author present suggestions for further study?
Do the stated conclusions flow logically from the obtained results?
++
Readers should find a clear statement of the authors' major conclusions based on their interpretation of the results and research hypotheses. The authors should compare and contrast results with other related work, to offer support for existing clinical theory or propose an alternative theory or explanation. The authors should acknowledge factors about the subjects, materials, or methods that could have complicated the interpretation of the results. As the authors discuss these limitations, they should share their ideas for approaching the research question differently. On the other hand, if the authors expressed confidence in the design and execution of the study and, therefore, have confidence in the results, they should suggest a direction that future studies might take. Finally, and perhaps most importantly, the authors should discuss the impact of the results on clinical practice. Is their evidence strong enough to suggest a need to change some aspect of treatment intervention or practice models? As critical consumers of the products of clinical research, readers should study the discussion and conclusion sections of the research report to decide whether the authors' answers to these kinds of questions are true, appropriate, and justified.