Chapter 34: Evaluating Research Reports

The link between research and clinical practice in health professions must be made by those who read reports of completed research. As consumers of research, we have a responsibility to evaluate research reports to determine whether the findings provide sufficient evidence to support the effectiveness of current practices or offer alternatives that will improve patient care. The success of evidence-based practice will depend on how well we incorporate research findings into our clinical judgments and treatment decisions.

For most of us, it is neither practical nor possible to read the plethora of material presented each month in professional journals. Therefore, the first step in effective reading is to select publications and articles that offer the most useful information. When a particular topic is important to practice, readers should begin by searching for systematic reviews that present an analysis of current knowledge on diagnostic methods or treatment effectiveness (see Chapter 16). Readers may also consider textbooks or traditional narrative review articles that provide basic, often in-depth, information on a topic; however, we must realize that the information is second-hand, presented from the perspective of the textbook or review article author. The problem is one of interpretation; the original research may be misrepresented or inadequately described. Therefore, consumers of research must be able to access the first-hand reports of researchers to judge the merits of their work.

Critical analysis of a research report is necessary to determine its validity and, thereby, its applicability for clinical decision making. The structure and content elements of research reports have been described in Chapter 33. The purpose of this chapter is to present a practical approach for critically reading and evaluating published literature, describing the kinds of questions readers should ask. Authors may also use this approach to appraise their own manuscripts before submitting them for publication.^∗ Although this chapter focuses on the written research report, these principles apply to the evaluation of oral and poster presentations as well. This evaluative process works for purposes of literature review, answering specific clinical questions for evidence-based practice, or just keeping up with progress in an area of interest.

We will focus this discussion on three areas: trials that test the efficacy of interventions, studies that measure the accuracy of diagnostic/screening tests, and studies that examine the predictive validity of variables for prognosis. Guidelines for evaluating systematic reviews were presented in Chapter 16.

One of the first considerations in evaluating the scientific merit of an article is the reputation of the journal. Scientific journals are generally refereed, which means that the articles have been subjected to review by content experts and accepted for publication on the basis of reviewer evaluation. Instructions to Authors, provided in most journals, will describe this process. The editors and reviewers of refereed journals follow policies and procedures designed to ensure that published articles meet defined criteria, including importance of the research, originality, appropriateness of design, adequacy of the method, soundness of conclusions and interpretation, relevance of the discussion, and clarity of writing. The manuscript review process is similar, in fact, to the way in which responsible readers will evaluate published articles.

Authors are usually asked to make revisions to improve the quality and clarity of a manuscript before it is accepted for publication. This process does not guarantee, however, that a study is without flaws. It remains the reader's responsibility to evaluate a study's validity and applicability. Important information may be presented in nonrefereed journals; however, readers must realize that the contents of those publications have not been scrutinized in the same way as those in refereed journals.

Clinicians and researchers will evaluate literature from three perspectives. First, the assessment will help to determine if the study has validity in terms of design and analysis. Second, the results of the study are examined to determine the importance of the effect of intervention, the accuracy of diagnostic tests, or the risks associated with prognostic variables. Finally, the clinician may consider how effectively the results can be applied to a particular patient's management.

Many useful references are available to guide the process of critical appraisal of published papers.^{2,3,4,5,6,7,8,9} Several of these guidelines are also used to assess methodological quality in systematic reviews (see Chapter 16).

What Is the Study About?

Before we can assess the validity of a study, we must first understand its intent. Researchers begin this process at the top, by reading the title and abstract. Titles should be informative, but are often so abbreviated that the reader is unable to learn much about content other than the general topic. If the reader is interested in the topic, then the next step is to read the abstract. Abstracts of research papers will include fairly specific information about the purpose, subjects, method, results, and major conclusions of the presented work. When readers decide that the results and conclusions stated in the abstract could be applied to their practice, they should read the body of the paper.

The introduction to an article tells us about the purpose of the study and how the authors developed the research question. It informs readers by providing a review of literature that frames the theoretical foundation for the study. The following questions help to focus the content of the introduction:

• What is the problem being investigated? Has it been clearly stated? Why is it important?

• How has the author used the literature to form a sound and logical rationale?

• What is the theoretical context for the study?

• Are references appropriate and comprehensive?

• What type of research does the study represent?

• What is the specific purpose of the study?

• What are hypotheses or guiding questions that form the basis of the study?

In the opening sentences of an article, the author should establish the problem being investigated. The background material should demonstrate that the researchers have thoughtfully and thoroughly synthesized the literature and related theoretical models. This synthesis should provide a rationale for pursuing this line of research. Readers should be convinced that the study was needed.

By the end of the introduction, the specific purpose or aims of the study should have been clearly stated and the research question should be evident. The authors of an intervention study should have explicitly stated the hypotheses that were to be tested. If not, however, readers should be able to determine what the researchers expected to find. For a diagnostic accuracy study, it should be clear if the intent is to compare two tests, to estimate the accuracy of one test, or to examine a test's utility across participant groups. Studies of prognosis should document the relevance of predictive variables and validity of outcomes.

Are the Results of the Study Valid?

The foundation of a research report is the method section, providing the details of the study. This is where we learn enough to decide if the results and conclusions of the study are meaningful. Flaws or omissions detected in the methods section affect the usefulness of interpretations derived from the study. Table 34.1 shows the questions that should be asked to assess the validity of an intervention study. Questions relevant to diagnostic accuracy and prognosis are shown in Tables 34.2 and 34.3.

Subjects

Readers must know who the subjects were in order to interpret the validity of a study's conclusions and to understand the extent to which the findings can be generalized or applied to specific patients. We can ask questions to determine whether sampling bias might have existed and to identify potentially confounding factors. Age, gender, diagnosis, comorbidities and level of function of the subjects are among the many factors that may affect the validity of findings. Inclusion and exclusion criteria should be specified so that the target population for the study is clearly identified.

The number of subjects in a study is a consideration when interpreting the results of the statistical analysis. Specifically, failure to demonstrate statistically significant effects should not be assumed to mean that no effect truly exists. The authors should present the results of a power analysis to determine the possibility of a Type II error resulting from an inadequate sample size.

For intervention studies, the description of subjects should include information about the accessible population and whether a probability or nonprobability sampling process was used. This information provides a foundation for deciding how well the findings can be generalized.

In diagnostic studies, the sample should reflect an appropriate spectrum of patients for whom the test would be applied. A diagnostic test is useful only to the extent it can identify those with varied degrees of the target disorder.¹⁰

For prognostic studies, authors should provide a complete description of the duration of the disease or disorder at the time patients entered the study.¹¹ Patients should be at a similar stage of disease to demonstrate consistency of outcomes.

Design

Based on the stated purpose of the project, readers should be able to judge the appropriateness of the design choices that were made. For studies of interventions, randomized designs are important for the control of internal validity, although quasi-experimental studies may still provide useful findings with sufficient description and controls. In diagnostic and prognostic studies, cohort designs are most useful for following subjects over time. Levels of evidence that are appropriate for these studies have been summarized in Chapter 16 (Table 16.1). It is important, however, to take these recommendations as guidelines only, understanding that the quality of the study must be examined to determine its applicability as evidence for clinical practice.

Information on study design will allow the reader to determine if the study is free from bias. As much as possible, researchers should incorporate blinding of subjects, testers, and investigators to protect against such bias. For intervention studies with two or more comparison groups, blinding provides some assurance that responses are not distorted because of inconsistent application of procedures or measurements. For diagnostic studies, those who administer and score tests should be blinded to the subject's true diagnosis. In prognostic studies, those who measure outcomes should be blinded to the subject's prior status.

The design will also provide a framework for understanding points of data collection, and the extent to which subjects are followed over time. For intervention studies, this has implications for understanding differences in trends and changes across groups. For prognosis studies, where the presence of a prognostic factor can precede outcomes by long periods, follow-up is most important to be sure that the study duration is long enough to detect the outcomes of interest.

The balance between efficacy and effectiveness should also be considered. When true experimental designs are employed, with the strict control inherent in these designs, readers should consider how to relate the findings to their "real world." In the "experimental" setting, where confounding factors and internal validity are rigorously controlled, the effects of manipulating or imposing the independent variable can be accepted with a high degree of confidence. But have these controls created such an "artificial" situation that it would be unrealistic to expect the same outcome when methods or procedures are implemented in clinical practice?

On the other hand, if quasi-experimental designs or descriptive methods are employed, readers must be alert to the possibility that extraneous variables have interfered with the interpretation of the results. The validity of an experimental or quasi- experimental study will depend in large part on how subjects are assigned to groups and whether or not a control group was included in the design. For example, when intact groups are used as in case-control designs, the author should specify the basis on which the subjects were selected.

Procedures

Readers should know what specific instruments or tools were used for measurements. If these are standard and commercially available, the model numbers and names and addresses of manufacturers should be included. If standardized questionnaires or survey instruments are used, references must be cited. When appropriate, reliability of measurements should be documented. Such documentation may be referenced from previous studies, or the researcher may establish reliability within the context of the study being reported.

The description of procedures should report the sequence of events from beginning to end. Description of data collection methods should include who performed the measurements, what the subjects were asked to do, what the data collectors did, and when and how often measurements were taken. All of these activities must be described in sufficient detail so that readers could, in a similar setting with similar subjects, replicate the study procedures. Studies of interventions typically involve comparison of groups or conditions, and the author should document the extent to which these groups were treated equally, except for the experimental treatment.

Data Analysis

Both descriptive and inferential statistical analyses should be identified. If inferential statistics were used, the acceptable significance level, such as α = .05, should have been established by the researchers and reported. The statistical tests should be described not merely by name, but by specifying to which data the analyses were applied. This information is necessary for readers to evaluate the appropriateness of the analyses. The use or misuse of statistics should be judged on the basis of two major factors: the nature of measurements (scale, reliability, linearity, and so on) and the study design (the number of groups or variables and frequency of measurement). Discrepancies in the proper application of statistical tests interfere with the statistical conclusion validity of the study and detract from the interpretation of the data.

Authors should report the degree of follow-up that was achieved and if attrition was present in the study sample. Reasons for attrition should be provided, to help determine if bias was present. In intervention studies, the author should specify if an intention to treat analysis was done, and how missing data were handled. The degree of follow-up is especially important for prognostic studies, to assure that outcomes are representative of the sample.

Are the Results Meaningful?

The results section of a research report should contain the findings of the study without interpretation or commentary. Readers should find a narrative description of results, typically including test statistic values, such as t or F, odds ratios, or likelihood ratios.

It is rarely sufficient to simply determine that values are "significant," however. The usefulness of a study's results lies in the size of the effect that was demonstrated. Effect size tells us about the strength of the observed relationships. As we strive for greater application of evidence in clinical practice, we must understand just how much change we should expect from an intervention, how accurate a diagnostic will be, or how likely prognostic outcomes will occur. Statistics such as risk ratios, likelihood ratios, and number needed to treat will help us understand these effects. The precision of these estimates should be reflected in confidence intervals.

The narrative description of findings may be illustrated with or complemented by figures and tables to clarify and summarize the characteristics of the data. Readers should study graphs and tables carefully. The effect size indices are often presented in tabular fashion, rather than in the text. It is a useful exercise to compare data from tables with information found in the text, to be sure that the discussion of results follows what was actually reported.

What Does It All Mean?

A research report ends with the authors' discussion and conclusions, putting the results of the study in context. The discussion section of an article should address each of the research questions, and should show how relevant literature helps to support the author's conclusions. Some useful questions to focus this part of the review include:

• How does the author interpret results?

• Did the author clarify if hypotheses were rejected or accepted?

• What alternative explanations does the author consider for the obtained findings?

• How are the findings related to prior reports?

• What limitations are described? Are there limitations that are not addressed?

• If results are not significant, does the author consider the possibility of Type II error?

• Regardless of the statistical outcome, are the results clinically important?

• Does the author discuss how the results apply to practice?

• Does the author present suggestions for further study?

• Do the stated conclusions flow logically from the obtained results?

Readers should find a clear statement of the authors' major conclusions based on their interpretation of the results and research hypotheses. The authors should compare and contrast results with other related work, to offer support for existing clinical theory or propose an alternative theory or explanation. The authors should acknowledge factors about the subjects, materials, or methods that could have complicated the interpretation of the results. As the authors discuss these limitations, they should share their ideas for approaching the research question differently. On the other hand, if the authors expressed confidence in the design and execution of the study and, therefore, have confidence in the results, they should suggest a direction that future studies might take. Finally, and perhaps most importantly, the authors should discuss the impact of the results on clinical practice. Is their evidence strong enough to suggest a need to change some aspect of treatment intervention or practice models? As critical consumers of the products of clinical research, readers should study the discussion and conclusion sections of the research report to decide whether the authors' answers to these kinds of questions are true, appropriate, and justified.

As we search for information to apply to our clinical decision making, we all benefit from applying literature to patient cases. A systematic review is one approach to critically understanding the quality of evidence on a topic (see Chapter 16). The systematic review, however, is based on an overview of the topic, not its application to a specific patient. An alternative format, called a critically appraised topic (CAT), has become popular to provide a brief summary of a search and critical appraisal of literature related to a focused clinical question.^{12,13,14,15,16}

A CAT provides a standardized format to present a critique of one or more articles and a statement of the clinical relevance of the results. A CAT is typically initiated by a patient encounter that reveals a knowledge gap.¹⁷ The author of a CAT searches for and appraises "current best evidence" from the literature, summarizes the evidence, integrates it with clinical expertise, and finally suggests how the information can be applied to a patient scenario.¹⁸ The critique will address internal, external, and statistical conclusion validity. Table 34.4 shows a sample CAT of an intervention study.

Format of a CAT

The format of a CAT can vary, although the essential elements remain fairly constant. It is generally a brief document, typically one to two pages in length. Information is intended to be concise. Table 34.4 shows one common layout. At minimum, a CAT should contain the following information:

• Title: A concise statement that will be used to catalogue the CAT.

• Author and Date: The author of the CAT should be specified, as well as the date the search was executed. A revision date should be proposed.

• Clinical Scenario: A concise description of the patient case that prompted the question.

• Clinical Question: This is the question that was developed from the patient case. It includes the elements of PICO (see Chapter 1): The patient/population, the intervention that is being considered (diagnostic test or prognostic variables), a comparison (if relevant), and the outcome of interest. This structure helps to refine the question and identifies key elements of an efficient database search.

• Clinical Bottom Line: A concise summary of how the results can be applied; a description of how results will affect clinical decisions or actions.

• Search History: Description of the search strategy used to obtain the studies that are being appraised (see Chapter 31). This includes databases used and terms entered at each step. Relevant studies are selected, often based on achieving highest levels of evidence (see Chapter 16).

• Citations: Full bibliographic citations of studies selected for review.

• Summary of the Study: Is the evidence in this study valid? This section should provide a description of the study based on the questions shown in Tables 34.1, 34.2, 34.3, including the type of study, subjects, procedures, and design elements.

• Summary of the Evidence: The results of the study should be summarized in narrative and/or tabular format. Specific effect size estimates should be provided as appropriate, such as means and mean differences, confidence intervals, odds ratios, likelihood ratios, sensitivity and specificity, and number needed to treat. If these estimates are not included in the research report, the CAT author may be able to calculate them if sufficient data are provided in the study.

• Additional Comments: Provide critical comments on the study, including issues related to the sampling, methods, data analysis, quality of discussion, and interpretation of results. This is where you should comment on the internal, external, and statistical validity of the study. Comment on positive and negative aspects of the study.

A CAT has two important distinguishing features for evidence-based practice. First, it is based on a specific clinical question, prompted by a clinician's need to clarify an aspect of patient care related to intervention, diagnosis, prognosis, or harm.¹³ This question is developed using the PICO format, as shown in Table 34.4. The clinician searches the literature explicitly to provide an answer to this question, with the intent of influencing management of the patient.

The second unique feature of a CAT is the inclusion of a clinical bottom line, or conclusion by the CAT author as to the value of the findings. The usefulness of the bottom line depends on the ability of the CAT author to accurately assess the validity of the literature and to grasp the relevance of findings for a particular patient's management.¹⁸ This information is then useful to others who may encounter similar patients, who can take advantage of the summary for efficient decision making.

Using CATs for Clinical Decision Making

The utility of CATs will depend on the clinician's ability to readily access the information within the clinical setting. Wyer¹⁴ suggests that CATs may serve to make the results of journal clubs or case conferences available to clinical staff to improve the care of subsequent patients. He offers the perspective that building such bridges and making them available at the point of care should be an essential part of evidence-based practice. Many institutions have established online "CAT Banks" that provide ready access to summaries of articles on specific topics.^{12,19,20,21,22,23,24}

CATs do have limitations as a source of clinical data. They have a short shelf life as new evidence becomes available, which is why revision dates are included. Clinicians must recognize that CATs do not represent a rigorous search of the literature, as in a systematic review. CATs are often based on only one or two references and may not represent the full scope of literature on a topic. Their usefulness is conditional on the critical appraisal skills and accuracy of the author. It is, therefore, the clinician's responsibility to determine the relevance of the clinical question, the logic of the search strategy, and the application of validity criteria in the review.²⁵ It is also important to assess the strength of the CAT author's conclusions in relation to the evidence provided. Notwithstanding these limitations, however, in the hands of discerning clinicians, CATs provide a practical method for disseminating evidence from the literature to inform clinical decisions.