Section III: Analyzing the Evidence in the Literature

INTRODUCTION

The third element in the effective integration of evidence-based principles into physical therapist practice is the use of scientifically derived evidence. Over the past two decades, studies suggest that health-care practitioners have difficulty changing their methods of practice from those they were taught in school and in enhancing their knowledge with newly developed scientific findings. Accessing and evaluating the quality of scientific evidence has become a challenge for busy practitioners. Each year brings new and better tools to help clinicians integrate evidence into their clinical practice.

Although there are numerous excellent textbooks available for physical therapists with an in-depth approach to accessing and evaluating scientific evidence, this section of our text will present a broader perspective on integrating evidence. We are strongly influenced in our framework for this section by the work of R. Brian Haynes, who has recommended a hierarchical model for organizing information of value to clinicians in making clinical decisions.^1,2 The pyramidal format (Fig. 1) is based on the relative value of various sources of evidence to clinicians. It is not based on the strength of the original evidence, such as the pyramid found in Chapter 11 of this section, but is based upon the value-added aspects that come with filtered and enhanced sources of evidence. Haynes' evidence pyramid acknowledges the value to the clinician when evidence from single studies has undergone secondary review and synthesis by experts.

Haynes recommends that clinicians work from the top down in this model, trying first to obtain the most limited but the most valuable of the evidence-based information sources available to them. In presenting the chapters in this section, we will work from the bottom of the pyramid, which is more familiar to most and where the volume of available resources is greatest, and then move upward.

Chapter 10 sets out the first step in using any source of evidence, asking a sound clinical question that is based on the needs of patients and the judgment of the clinician. These questions guide the search for new knowledge about the diagnosis, prognosis, or treatment of patients. Knowing what you are looking for in the evidence will enhance the likelihood that the evidence you find will fit your purpose. In Chapter 11, Connie Schard discusses current resources and methods to access unfiltered evidence contained in original scientific studies. The chapter also describes access to databases of filtered syntheses, synopses, and summaries. Chapter 12 reviews the value of original experimental research by highlighting the methodological and statistical principles that represent high-quality research. Chapter 13 provides a set of questions to use in assessing individual studies for their utility in guiding care. In Chapter 14 we address a variety of evidence resources both in physical therapy and medicine that are synopses of original studies and discuss the strengths of each type of synopsis. In Chapter 15, Alison Hallam, an experienced physical therapist reviewer for the Cochrane Collection, moves up the pyramid to syntheses in the form of systematic reviews or meta-analyses, discussing how the clinician might read and evaluate this very useful form of evidence for practice. She also includes a discussion of the synopses of syntheses. In Chapter 16, we describe Haynes's recommendation to produce summaries as evidence that integrates data on a specific topic from the previous four levels of the model, often in the form of practice guidelines. This chapter also provides a discussion of how evidence from physical therapy practice can be integrated with scientific evidence in point-of-care systems, which can be integrated into electronic record systems.

By exploring each of these sources of evidences, we provide many tools with which to find and assess the evidence that is available to help you improve care for your patients.

REFERENCES

INTRODUCTION

We have always had questions like these about patients like Mr. Ketterman. In the past, we have turned to a variety of sources for answers, including:

• Tradition: This is the way it's always been done. Tradition has been handed to us by our academic and clinical teachers as well as by role models and mentors.

• Authority: This is the way I'm told to do it. Again, our teachers sometimes tell us to do things in certain ways. We may also be told to do them in a certain way by our clinical supervisors or employers.

• Trial and error: This seems to work before, let's see if it will work now. Sometimes we try to organize the responses to our trial and error to learn from them, but often trial and error just happens as we search for solutions in our practice. As we learned in Section I, our memories of what worked may not be very reliable.

None of these sources is very satisfactory as a way to really improve patient care. Instead, it would be much better to build decisions on the research that has been done about care. In the past, this task was quite daunting due to the vast amount of information and a lack of a framework for assessing the information. But the principles of evidence based practice provide an organized and systematic way to ask and answer these kinds of questions.

Asking the Question

The very first step of evidence based practice is to convert clinical questions like the ones about Mr. Ketterman into questions whose answers can be sought in the literature. The first step in that process is to recognize that different kinds of questions will be answered in different ways. The first two questions, those about Mr. Ketterman's diagnosis, are what have been called background questions.

Background Questions

Background questions focus on learning more about the typical course or natural history of a pathology or trauma. Background questions also focus on the typical management of a patient problem and the typical responses to this management. Clinicians tend to ask background questions when they are new to a particular patient population. For students and new graduates this can be almost all the time, since they have not yet had a chance to become familiar with the course of disease or disability across a patient's life or episode of care. Clinicians who have experience with a particular patient population may also want to ask background questions to identify new knowledge, such as advances in what is known about etiology, life expectancy, or changes in typical management. But an experienced clinician can also ask background questions when faced with a patient from a population that is unfamiliar.

Foreground Questions

Once clinicians have an understanding of the typical course of events, they can begin to ask more specific questions having to do with the particular care being provided to this patient. These are called foreground questions. Questions 4 and 5 in Mr. Ketterman's case study are examples of foreground questions. These are questions asked to help choose tests and measures for diagnosis and prognosis or to choose interventions. Foreground questions apply both to management of known problems and to prevention and wellness. All clinicians need to ask foreground questions all the time since the evidence about the better choices in care is constantly changing. But they can't stop and ask questions about everything they do. As discussed in Section I, clinicians need to use clinical judgment to identify those topics that matter most. So, clinicians can focus foreground questions on patient issues that represent a significant portion of care, on aspects of care that pose an increased risk for the patient, or on aspects where they know the evidence is changing rapidly.

Figure 10-1 shows the relationship between background and foreground information. As clinicians gain in experience and knowledge about particular patient issues, their background questions about exactly what the problem is will diminish, to be replaced by increasing foreground questions to help them improve their diagnostic ability or intervention choices. The two triangles extend out beyond each other to show that even when the questions are primarily of one type, there are still questions of the other type.

Clarifying whether the question is background or foreground can immediately lead to determining how and where to search for answers. Background information is more typically found in textbooks and clinical updates; foreground information is more typically found in the evidence from research. In either case, we want to ask the best question that we can in order to find the best answers. Generally, this means being as precise as possible, without being so specific that no answers can be found. The questions arise from the patient in front of the clinician and therefore will almost always be quite specific as they are first formulated. What is happening to this patient? How can I help this patient improve? But, of course, that is not how the literature is organized. So the question must be changed to something like: What typically happens to patients like this one? What typically helps patients like this one improve?

Structure of Clinical Questions

Because it is the structure of the question that will lead to the answers, crafting the best questions possible is key. Many people working in the evidence based practice world recommend a particular format for formulating foreground questions about interventions, known as PICO questions.^1,2 A PICO question has four parts: the Problem or Population being addressed, the Intervention being considered, a Comparison intervention, and the Outcome that is of interest.

In formulating a specific PICO question, the P (problem or population) can be first developed by considering the patient's chief complaint or by generalizing the patient's condition to a larger population. In Mr. Ketterman's case, it would be: In patients with congestive heart failure.… As you develop the P in your question it is helpful to ask:

• How could you describe a group with a similar problem?

• How you would describe the patient to a colleague?

• What are the important characteristics of this patient?

  • Disease or health status

  • Age, race, sex, previous ailments, current medications

• Should these characteristics be considered as I search for evidence?

As the answers to these questions become more specific, the P part of the PICO question might become: In patients over 65 with congestive heart failure…. You might be tempted to ask: In white male patients over 65 with congestive heart failure who take Lasix to control their disease and who also have hypothyroidism…. But by providing this much detail, it becomes almost impossible to find relevant research, so you must strike a balance between precision and too much detail.

The second part of the PICO question concerns information sought about the Intervention. It's really important to recognize that this is based on the plan for the patient. If what you really want to ask is What can I do for this patient?, then you should consider this a background question and become more familiar with the specific options available for the patient's care and then focus a PICO question on one particular option. The options in a PICO are generally about interventions, but PICO questions can be asked about diagnostic or prognostic tests as well. In Mr. Ketterman's case, you might want to ask about the value of exercise for him. But simply asking about all exercise may well be too broad. You might focus on aerobic exercise or strengthening exercises. The choice we make will most likely be related to the outcome(s) in which we are interested, as we will discuss below. Avoid search terms that are actually practice-Specific but not used by researchers in describing their work. For example, physical therapists use "short arc quads" to describe a specific type of exercise, but this is not a searchable term in most databases. As is discussed in Chapter 11, many databases of health care research are organized using Medical Subject Headings (MeSH) terms. It is useful to spend some time with each database to learn the keywords or terms used by that database to organize the articles included in it.

The next part of the PICO question is the Comparison intervention to be explored. Many times clinicians need to make a choice between two different interventions. For example, they often wonder whether a new intervention might be superior to the existing one. In this case, the existing one becomes the comparison. Or they might want to consider two different approaches to reaching the same outcome. As with the primary intervention, the comparison intervention should be specific. In Mr. Ketterman's case, we may actually need to choose between aerobic and strengthening exercises and therefore want to understand the ability of each type of exercise to reach a certain outcome in patients like Mr. Ketterman. Choosing several multiple comparison interventions will most likely result in a less effective computer search, so comparisons are usually limited to one. The Comparison is the only component in the PICO question that can be omitted, as clinicians sometimes choose to look at the Intervention without exploring alternatives, and in some cases, there may not be an alternative. When there is no comparison of interest, the question is sometimes called a three-part question, that is, one that identifies:

1. the patient population,

2. the intervention in question, and

3. the outcomes of interest.

The final part to the PICO question is the Outcome of interest. The outcome specifies the result(s) of what the clinician plans to accomplish, improve, or affect and should be measurable. The outcomes may consist of things such as relieving or eliminating specific symptoms at the impairment level or improving or maintaining function. specific outcomes will yield better search results and allow you to find the studies that focus on the actual outcomes that matter for your patient. When defining the outcome, a term such as more effective is not specific enough; rather, you should describe how the intervention is more effective. For example, you could say more effective in increasing flexibility or in decreasing contractures.

So, we might ask a question like this to help us in treating Mr. Ketterman: In patients with Alzheimer's Disease with functional limitations related to deconditioning (P), will an impairment-based exercise program for endurance (I) versus a planned increase in daily activities (C) be more effective in restoring function (O)?

Questions About Diagnosis, Prognosis, and Outcomes

The examples used so far in developing sound clinical questions have all been related to choosing interventions for patients, but clinicians usually have questions across the patient care management model. In the areas of diagnosis, prognosis, and outcomes, they most often ask questions about whether a specific test or tool is the best available to help make a particular clinical decision. In Mr. Ketterman's case, we might want to know the best available tests to monitor Mr. Ketterman's ongoing cardiac status to use in his home, or what test will best help determine the progress made in his care. Both the PICO and the three-part question format can be used to help formulate these questions. The P remains the same—we always specify the patient problem or population of interest. Intervention and Comparison are used to refer not only to treatment intervention but to tests and measures. The Outcomes are framed in terms of the ability of the test to answer the clinical question. For example, we might ask an impairment-based diagnosis question such as: In patients with congestive heart failure is the 6-minute walk test a valid measure of impaired aerobic capacity? Or we might ask a prognosis question such as: In patients with Alzheimer's Disease and gait limitations, does the Berg balance test accurately predict the risk of falls?

Choosing the Answers

Once the question is clear, the next step in identifying the evidence needed to guide care is to decide what kind of resources will provide the necessary answers. Chapter 11 provides much more detail about how and where to seek answers. Chapter 12 will provide guidance on how to assess the literature once it is found. This chapter will focus on the type of literature we would want to find if we could.

Trustworthiness of Sources of Evidence

It is important to consider the varying levels of trustworthiness of the many sources of literature. Trustworthiness of the research that makes up the evidence can be described from a design or statistical point of view (see Chapter 12), but here we mean simply, what do I know about the sources that present the evidence to me? The materials considered to have the highest level of trustworthiness are those articles that have been subjected to masked peer review. This means that experts in the field have reviewed the authors' report of their research methods and findings and have found them acceptable for publication, without knowing who the authors were. Many journals are now adding to trustworthiness by requiring authors to indicate if they had any monetary support from entities related to the research question being studied. At the other end of the continuum would be claims made by manufacturers or sellers of a product. We are all familiar with the claims made in advertising for things like diet supplements that rely on testimonials that may not even come from actual people, but are just advertising claims. Between these two extremes of trustworthiness are a host of sources for information. Figure 10-2 lists some of these sources in declining order of trustworthiness. Unfortunately, as the potential trustworthiness declines, the ease of access increases.

As you search for information, there are many questions to ask, including:

• Is the material a report of research or is it opinion?

• What are the credentials and the motivations of the authors?

• What are the credentials and the motivations of the publishers?

When at all possible begin your search for answers with peer-reviewed information.

Types of Peer-Reviewed Research

One important factor to consider is the type of research needed to answer specific kinds of questions. This factor is so important that some people say the question format should be PICOD, where the D stands for design.³ There are two overarching types of research design that can be explored to find answers to questions. One type of research is termed quantitative research, primarily because it uses numbers to measure the activity being studied. But sometimes, we really need to understand the underlying phenomenon, and for that we turn to qualitative research, which typically uses words to describe what is being studied.

But these two types of research also have a more fundamental distinction than simply whether they use words or numbers. Quantitative research is also known as positivistic research since it is rooted in the concept that there is one certain or positive answer for a question and that by careful construction of experiments, one truthful answer can be found. Qualitative research is also known as phenomenology, because it is based on the belief that any given phenomenon, especially a human phenomenon, has many aspects, all valid and all truthful to the person experiencing the phenomenon. (See Chapter 8 for a description of qualitative research and ways to assess its trustworthiness, and Chapter 12 for a deeper description of quantitative research.) In health care clinicians often need to learn from both types of research.

Many questions can best be answered by various forms of quantitative research (Fig. 10-3). Questions about the accuracy of diagnostic tests can best be answered by seeking studies that are cross-sectional. These are studies that include subjects who are known to have a variety of diagnoses that all possess some similarities and among which we want to differentiate. The researchers then administer the test to see if it was able to correctly differentiate patients with different diagnoses.

If the question is about the tests used to help us make prognostic decisions about patients, clinicians would want to find reports of studies that are longitudinal in nature. The only way to tell if a test accurately predicts a future event is to follow patients for an extended period after the test has been administered to see if the test results were correct.

Questions about the usefulness of an intervention are answered by studies that are experimental. These types of studies use controlled trials to compare the results of a given intervention for the experimental group against the results in a control group that did not receive the intervention. The many aspects of experimental research that need to be considered in assessing literature will be discussed in much greater depth in Chapter 12, but it is important to recognize these major differences in types of studies as you search out literature to help improve care. There is also an increased emphasis on using well-designed descriptive research to understand interventions. These designs allow the gathering of data from actual clinical practice, when structured experiments may not be possible. These approaches are also discussed in detail in Chapter 12.

All of these types of studies—cross-sectional, longitudinal, and experimental—are quantitative or positivistic. But sometimes questions concern what it actually means to a person to live with a particular disease or level of disability or why a person might choose to accept or reject a particular intervention against the advice of the therapist. In this case, the clinician would turn to qualitative or phenomenological research, as it allows exploration of the perceptions and beliefs (see Fig. 10-3 and Chapters 8 and 12).

Types of Evidence

As we discussed in the introduction, evidence in the literature comes in many forms, as shown in Figure 10-3.

Single Study

The individual study is the most easily identified type of evidence used to help answer questions. When first beginning the work of seeking answers to questions, clinicians sometimes assume that somewhere out there will be the study that exactly answers their specific question about a specific patient. They also may assume that the study will be perfectly designed so that they can have complete faith in its results and that the results of the study are so powerful that they can have no doubt about what the results mean for the patient. But as the searcher progresses, it is soon found that this perfect study is ephemeral and elusive, always somewhere just out of grasp. Individual studies, no matter how well done, can seldom meet all of the desired qualities. This means studies must be analyzed carefully to determine their value in answering a particular question with a particular patient; in other words, are their results trustworthy (can we believe them) and generalizable (can we apply them to this specific patient)? Chapter 12 will cover this in some detail. Despite the difficulties in seeking answers study by study, this remains the most common way to find evidence, since individual studies are by far the most prevalent source of evidence. In fact, the sheer volume of individual studies can sometimes make a search seem endless. Chapter 11 suggests ways of searching with greater focus to help sift through the volume.

As Haynes has discussed,^4,5 if the information from these individual studies can be combined into more integrative forms of evidence, the job of the clinician becomes easier. He has described six levels of organization of evidence (Fig. 10-4), beginning with the single study. (Assessing single studies for their value in answering questions is covered in Chapter 12.) As mentioned, searching for answers single study by single study poses several problems, including their sheer numbers and the rigor needed to determine their usefulness. But for clinicians, one of the biggest problems may be that single studies tend to focus on only a single aspect of patient management, while the clinician must balance all of the patent's diagnostic and intervention needs at once.

Synopsis

The next level Haynes identifies in describing sources of information about evidence is the synopsis. A synopsis provides information about choices that can be determined from single studies. Synopses usually start or end with a clear statement about doing or not doing a specific thing with a specific patient population. The synopsis would also include the essential information from the study or systematic review that supports this conclusion. When appropriate synopses are available, a clinician can determine a course of action in a matter of a few minutes. But it is very important that the clinician have a clear understanding that the study really does match the clinical question. Also, the problem remains that the synopsis focuses on one aspect of patient management rather than on the range of decisions needed in actual care. Synopses are sometimes written in the Critically Appraised Topics format.¹ The Clinical Bottom Line feature found in Physical Therapy is another example of a synopsis format. Chapter 13 discusses the creation and use of synopses in more detail.

Synthesis and Synopses of Synthesis

Haynes suggests that the next level of sources of evidence is synthesis. Systematic reviews are the most common example of syntheses. These reviews are based on extensive literature searches designed to find as much of the research about a topic as possible, an analysis of each article found, and then an integration of the findings of the studies. The reviews have narrative assessments of each study and often rate their quality. Many systematic reviews also include meta-analyses, which are statistical analyses that allow the results of multiple studies to be merged, thereby giving more strength to the results. Many reviews also have some visual analysis of the results, such as forest plots, that allow the reader to see the ways in which the studies have both similar and varying results. The Cochrane Collective has led the way in publishing systematic reviews, but reviews are also appearing with increasing frequency in other journals. Chapter 14 discusses systematic reviews in more depth. The next level of sources is synopses of syntheses. These sources provide clinicians reviews of systematic reviews. They are also discussed in Chapter 14.

Summaries

The next level of sources of information about evidence is summaries. Summaries are designed to move past the basic problem of the previous three levels. They seek to integrate findings across the full management of a patient. They offer comparisons on a full range of evidence about the multiple interventions available for a given problem. Summaries about treatment options might be found in well-designed, evidence-based textbooks. In addition, clinical decision rules or clinical prediction rules also provide summary information to help clinicians compare the utility of various diagnostic tests in actually predicting the success of particular interventions. Summaries are very helpful because they much more closely approximate the actual decisions clinicians need to make, but they are, at the same time, much harder to find. Chapter 15 discusses the creation and use of summaries.

Systems

Haynes has placed systems at the top of his pyramid as a source of information about evidence. Systems allow full integration of evidence in the literature with the clinician's decision making at the point of care with the patient. To make the information available at the point of care, systems are often integrated into electronic medical records (EMR). As clinicians enter their patients' data in real time into the EMR, the system can provide clinicians with on-the-spot access to summaries and synopses, prompt them with the next option point, query them about their decisions if they seem to fall outside the parameters of the evidence, and provide information to the clinician. If the strength of the evidence supports it, systems can even be designed to deter the clinician from making a poor choice. They generally don't prohibit the clinician's decision, but instead prompt the clinician to reconsider by making queries and providing information. Systems are the best way for clinicians to access and use evidence in the course of everyday care, since the results are available almost instantly and the clinician does not have to stop to search for other data. However, as you might have suspected, there are very few examples of functioning systems, especially in physical therapy. Chapter 15 also talks in more depth about the creation and use of systems.

We have presented these six levels by moving up Haynes's pyramid, since this is often the way clinicians think of the search process and it mirrors the availability of different sources of evidence. Haynes advises, however, that clinicians search down the pyramid, since finding information at a higher level has so much more utility.

CONCLUSION

This chapter started with a description of the process of writing a good clinical question, as the question drives everything else we do in seeking answers. We identified some concerns in identifying the type of literature that we need to seek. The goal should be to search for literature that has been peer reviewed, whose design matches the type of question we want to answer, and that is at the highest possible level of source of evidence. Chapter 11 will give more detail on how to go about actually finding such research. But it is essential to keep in mind that finding useful evidence for clinical decisions may still be difficult. Chapter 12 will talk about some of the ways you can determine just how good the evidence is and if it is good enough. Clinicians cannot wait for the "perfect" evidence; they need to act and act on the best evidence they can find. Section IV will discuss in greater detail how clinicians can integrate the best evidence they can find with their own sound decision making and with patients' preferences to make the best decisions possible in the real world of clinical care.

SELF-ASSESSMENT

1. Refer to Mr. Ketterman's case in the Appendix and write three background questions about things that puzzle you about his medical status.

2. Refer to Mr. Ketterman's case in the Appendix and write three PICO questions that will help you choose diagnostic, prognostic, or outcome measurements to use in planning his care.

3. Refer to Mr. Ketterman's case in the Appendix and write two more PICO questions that will help you choose interventions for his care.

CONTINUED LEARNING

• Based on one of the PICO questions you wrote about Mr. Ketterman, or any other PICO question of interest to you, identify an article in the peer-reviewed literature that addresses the question. Then, using an Internet search or other sources, find an article written by a vendor or by a consumer group.

1. What differences do you see in these two types of information?

2. How can you help patients see these differences and make wise choices about their care?

REFERENCES

INTRODUCTION

Finding answers to clinical questions can be a rewarding activity once you become familiar with the resources available to you and understand effective searching techniques. This chapter will focus on finding the evidence to clinical questions, where it is stored and how you can efficiently retrieve it. We will also suggest methods to manage the retrieved references and discuss how to keep current with new information.

As we discussed in Chapter 10, questions that occur during the care of patients usually fall into one of two broad categories: background questions and foreground questions. Background questions are concerned with general knowledge that helps us to understand the process of disease. According to Strauss,¹ these questions have two parts: a root question such as "what" or "how" or "why" and an aspect of the disease or condition in question. These types of questions are usually answered with textbooks that provide general descriptions, overviews, and summary information. Foreground questions, often framed as PICO questions, are concerned with the management issues of individual patients with specific problems. Evidence based practice is centered on the care of specific patients and therefore provides a framework for addressing these foreground questions. Foreground questions are usually answered by current research, either in the form of filtered resources that select, analyze, and summarize studies as systematic reviews and preappraised topics or by primary studies that are reported in journals as original research, found through unfiltered research.

FILTERED RESOURCES

Filtered or preappraised resources have undergone a process of critical appraisal that systematically assesses the study for its methodology and validity, its results, and its relevance to making an informed clinical decision. Evidence-based practice requires the availability of current filtered evidence about diagnosis, treatment, and prevention of health disorders. Filtered evidence is characterized by a process of critical appraisal that evaluates the study for soundness of methodology, significance of results, and applicability to patients. The quality of the evidence is often visualized in the evidence pyramid, which shows the organization of resources based on the quality and comprehensiveness of the evidence (Fig. 11-1).

Filtered information can be quick to search and is validated for sound methodology. Examples of filtered resources relevant to physical therapy include: PEDro (Physiotherapy Evidence Database), Cochrane Database of Systematic Reviews, and the National Guideline Clearinghouse. (See Chapters 13, 14, 15, 16 for detailed descriptions of these forms of filtered information.)

PEDro

PEDro Physiotherapy Evidence Database (www.pedro.fhs.usyd.edu.au/) is an initiative of the Centre for Evidence Based Physiotherapy (CEBP) at the University of Sydney. It was developed to give rapid access to references and abstracts of randomized controlled trials, systematic reviews, and evidence-based clinical practice guidelines in physiotherapy. Most randomized controlled trials in PEDro have been rated for quality to help discriminate between trials that are likely to be valid and interpretable and those that are not. Searching is by words in the title or abstract and by broad categories of therapy, problem, body part, or study methodology. The results are displayed by type of study: systematic review, clinical trial, and practice guideline. The clinical trials include a critical appraisal score, which is documented in the full record. The database is small, with only about 19,000 records as of April 2011; it is updated monthly and focuses exclusively on selected relevant studies in physical therapy.

Cochrane Database of Systematic Reviews

The Cochrane Collaboration is an international, not-for-profit organization providing up-to-date information about the effectiveness of health care. The Cochrane Library (www.thecochranelibrary.com/) is a collection of databases that contain high-quality, independent evidence to inform health care decision making. There are several databases within the Cochrane Library. The one most relevant to physical therapy is the Database of Systematic Reviews. Cochrane systematic reviews represent a thorough review of the literature on a single topic, with conclusions based on high-quality studies that have been evaluated for validity, significance of the results, and applicability to patient care. There are several Cochrane Review Groups, including the Back Group, Bone, Joint and Muscle Trauma Group, and Musculoskeletal Group, which have developed systematic reviews on topics that are of interest to physical therapists. The full text of the Cochrane systematic reviews is only available online through a subscription service. Searching is by words in the title or abstract or by Medical Subject Headings (MeSH). The Cochrane Collaboration provides free access to the abstracts of their systematic reviews from their Web site at www.cochrane.org/reviews/index.htm.

The National Guideline Clearinghouse

The National Guideline Clearinghouse (www.guideline.gov) is a free, comprehensive database of evidence based clinical practice guidelines. Criteria for inclusion in the database include:

1. systematically developed statements that include recommendations, strategies, or information that assists physicians and/or other health care practitioners and patients make decisions about appropriate health care;

2. production under the auspices of medical specialty associations, agency, or organization; and

3. documented systematic literature search and review of existing scientific evidence published in peer-reviewed journals; and

4. availability of the full text of a guideline that was developed and reviewed within the last 5 years.

Searching is by text words and by browsing disease categories.

UNFILTERED RESOURCES

Unfiltered information usually requires a more focused search strategy and the need to critically appraise the study before applying the results to a patient. The unfiltered resources are often much larger and more comprehensive databases, such as PubMed and CINAHL (Cumulative Index to Nursing and Allied Health Literature).

Medline/PubMed

Medline is the U.S. National Library of Medicine's (NLM) premier database; it contains over 17 million references to journal articles in the life sciences with a concentration on biomedicine. Biomedicine includes the areas of behavioral sciences, chemical sciences, and bioengineering as they relate to basic research and clinical care, public health, health policy development, and related educational activities. PubMed is NLM's search service, which provides access to MEDLINE and the other databases of the National Center for Biotechnology Information (NCBI). (There are other search services, such as Ovid, Inc and MDConsult, which also provide a search interface to the Medline database.) A distinctive feature of Medline is that each reference is read by a subject expert who then indexes or tags the article using standardized terminology called Medical Subject Headings or MeSH. MeSH helps organize and identify articles on specific topics within the database. Due to its enormous size, searching requires a well-thought-out search strategy that includes subject headings and text words that can then be limited to specific parameters. While there is a growing number of free full-text articles linked from PubMed, to obtain the full text of most articles requires personal subscription or access to a medical library.

CINAHL (Cumulative Index to Nursing and Allied Health Literature)

The CINAHL database from EBSCO indexes over 3000 journal titles in nursing, allied health (including physical therapy), biomedicine, complementary medicine, and consumer health information from 1982 to the present. It includes approximately 2,300,000 records for articles, books, pamphlets, dissertations, government publications, audiovisuals, software, and proceedings. CINAHL is an indexed database, using CINAHL Subject Headings, many of which are similar to MeSH. Unlike PubMed, CINAHL requires a subscription and is therefore usually available only through medical libraries or professional associations. It is important to note that while these databases are the primary resources for finding journal articles, they may not index all the journals in the field of physical therapy and rehabilitation medicine. Several studies have shown that PubMed is the more comprehensive database for physical therapy journals, followed by PEDro and CINAHL.^2,3 Thus a thorough literature review may require searching all of these resources and databases. See Table 11.1 for a summary of these databases.

Tutorials

Most databases will provide tutorials or additional help with understanding the search techniques related to the database. Medical libraries also develop their own tutorials and search aids. Some examples:

• PEDro: http://pedro.org.au/english/search-help/

• PubMed: http://nlm.nih.gov/bsd/disted/pubmed.html

• http://mclibrary.duke.edu/training/pubmed

• CINAHL: http://support.epnet.com/training/tutorials.php

• http://mclibrary.duke.edu/training/cinahlebsco

THE SEARCH STRATEGY

Before you start your search for relevant information, you need to have a clear understanding of what you are looking for (the clinical question) and where you might find it (the resources). The basic steps of conducting a literature search are shown in Figure 11-2.

Focus the Question

Determine the important concepts that need to be present in the article or information you find. The PICO framework, discussed in Chapter 10, provides a model for developing a well-focused question. A well-defined clinical question can lead to a well-focused search strategy. It will also be useful in identifying the most relevant articles that address the specific question and patient.

Select the Best Resource

The well-formed clinical question will help guide your selection of the resources most likely to provide useful results. PEDro and Cochrane are considered filtered resources because the studies included in these small, specialized databases have been critically appraised and therefore provide some assurance that the studies are valid and the results reflect sound scientific methodology. PubMed and CINAHL are much broader in scope but do not evaluate the quality of the studies. Additional work and time will be needed to determine the validity and significance of the results of studies found in these databases.

Formulate the Search Strategy

There are several steps in formulating the search strategy:

• Separate the concepts:

  After selecting the most appropriate resources, you need to be familiar with how each resource is constructed and therefore the best way to retrieve relevant information from it. Use the concepts of your clinical question to form your search strategy.

• Select index terms:

  Medline and CINAHL use standardized terminology (called MeSH and CINAHL Subject Headings) that helps the researcher find all articles on a topic regardless of the exact words used by the author. This is especially helpful when there may be several ways to describe the same concept. For example, articles that discuss using electrodes to deliver pain relief may refer to this as "TENS" or "electroanalgesia" or "Transdermal Electrostimulation." However, the index term or subject heading in Medline and CINAHL is "Transcutaneous Electric Nerve Stimulation" and therefore all articles that discuss this concept will be indexed under this term. These index terms or subject headings are relatively easy to find. Most databases will provide a list of them (in Medline this is the MeSH Database; in CINAHL it is the CINAHL Subject Headings) or will automatically "map" (or translate) your search term to an appropriate subject heading.

• Select text words:

  Text words, on the other hand, are the words that the authors used in the title or abstract. Common problems with text words are that you need to include all the synonyms for a concept in a comprehensive search strategy, and while the word might appear in the abstract that does not guarantee the article is about it. (For example, "TENS" may appear in the abstract because the author stated that "the patients were not given TENS therapy….") Text words are useful when the concept has no equivalent subject heading, is a unique word or phrase, or is a new concept that has not yet been assigned a subject heading. Often the most effective search strategy uses a combination of index terms and text words to retrieve as much useful information as possible.

• Combine concepts:

  Search terms can be combined using the Boolean logic of "OR" to broaden or "AND" to narrow the retrieval. Using the logic of AND requires that all terms selected are in the same citation, whereas using the logic of "OR" requires that at least one of the selected terms is in a citation. These combining terms or Boolean logic are a universal language for all databases and search engines and must be applied correctly to get the best results (Fig. 11-3).

• Apply limits:

  Depending on the question and the amount of information retrieved from your search of the topic, you may want to limit the final results to a specific age group, language, type of study, or years of publication. Each database has its own set of limits that can be applied to further refine the search. PubMed offers a unique method for limiting retrieval to the most appropriate study methodology based on the type of clinical question being asked. Once you have a set of articles that address your topic, copy your final set number with the # sign into Clinical Queries, select the type of question category (therapy, diagnosis, prognosis, etiology, or clinical prediction), and select a narrow or broad strategy. The system will automatically add a short search strategy proven to get at the most appropriate study methodology based on the type of question selected.

Review the Results

Review the titles and abstracts for relevance to the clinical question. If they are not relevant to your question, you may need to modify your search strategy, refocus your question, or select another resource. Searching is an iterative process that may take several attempts before reaching a satisfactory retrieval. One way to improve your search strategy is to identify a highly relevant citation and then view the complete record for the indexing terms assigned to it (Table 11.2). This may give you suggestions for additional terms to use in refining your search strategy. If the results are relevant, then they should be evaluated to determine if they can be applied to patient care. (See Chapter 12.)

SEARCH ENGINES

Search engines are computer programs designed to help find information on the public Web. They look for content matching specific criteria, usually words or phrases, and retrieve a list of items that contain the exact words or character strings. The items are usually displayed in a ranking that may depend on the frequency of the words and their location in the page or document rather than their relevance to the topic.⁴ Google Scholar is a subset of Google that searches across disciplines and published sources, including peer-reviewed papers, theses, books, abstracts, and articles, from academic publishers, professional societies, preprint repositories, universities, and other scholarly organizations. The results are ranked according to a number of criteria such as full text, number of times cited by other authors, and the publication in which the article appears. It is important to note that although Google Scholar is a good way to limit your search in Google to the published literature, there is no way to know exactly what journals were included in the search.⁵

There are some specialized search engines that search across a well-defined subset of EBM-related Web sites, such as TRIP (Turning Research Into Practice). The purpose of the TRIP Database (www.tripdatabase.com) is to allow health professionals easy access to the highest-quality material available on the Web to support evidence based practice. TRIP, based in the UK, searches across over 90 Web sites of high-quality medical information, providing direct access to abstracts from journals (such as the BMJ, JAMA, and NEJM), practice guidelines, textbooks, (such as eMedicine and the Merck Manual), systematic reviews, and evidence based synopses (such as POEMS, Bandolier, and BestBets). In addition, TRIP provides a link to conduct the search in PubMed using the Clinical Queries feature.

Because search engines query across many different Web sites, you can only search by text words or words that appear on the Web page. This means that you may have to try several different word combinations or synonyms before you find exactly what you need. It also may result in retrieving a very large number of items, many of which may not be relevant. Search engines such as Google will include all types of Web sites, including commercial, personal, governmental, and educational, making it the responsibility of the searcher to figure out which sites provide the most authoritative, unbiased, and accurate information.

ACCESS TO RESOURCES

One common misconception about the resources is that "everything is available for free on the Internet." Although some of the resources may be free because they are older material or provide limited access to just abstracts, getting to authoritative, full-text information often requires subscription fees and access to a medical library. The Internet is the communication platform that allows libraries and health care professionals, as well as businesses, to provide access to databases and products for their patrons, customers, or audience. The resources listed in this chapter are usually available at most medical centers or community hospitals with library services. A medical librarian can help you access these databases, provide advice on the most efficient search strategies, and help obtain copies of relevant articles.

If you do not have access to a medical library, there may be other ways to access some of these resources. The National Library of Medicine coordinates the National Network of Libraries of Medicine (NN/LM) (http://nnlm.gov), whose mission is to provide health professionals and the general public with health information resources and services. They do this through outreach programs and an extensive network of hospital and academic medical libraries. Area Health Education Centers (AHEC) are also involved in supporting education and training for health care professionals. These regional and statewide programs support health libraries and coordinate access to information services and resources. (www.nationalahec.org/)

Resources Available From the American Physical Therapy Association (APTA)

The American Physical Therapy Association (APTA) (www.apta.org) offers a members-only service that provides access to CINAHL, the Cochrane Library, and selected research journals through the association Web site. The Open Door: APTA's Portal to Evidence-Based Practice provides members with easy access to journals and other resources relevant to clinical practice whenever and wherever they need it. In addition, the Open Door maintains a list of current research articles and an e-mail alert system to keep current on new additions and searching tips for the Open Door project. Another service of APTA is the Hooked on Evidence Web site. This represents a "grassroots," member-generated effort to develop a database containing current research evidence on the effectiveness of physical therapy interventions. To be included in the database, studies must involve human subjects, examine at least one physical therapy intervention, report outcomes, and be published in an English-language, peer-reviewed journal. This service is discussed in greater detail in Chapter 14.

MANAGING THE EVIDENCE

Once you have retrieved a set of citations or references from a database or Web site, you may need a way to organize the items so that you can review and use them at a later time. This is especially true if you are conducting ongoing research, writing a paper, or require more time to review the material. Most of the large database systems offer advanced features that provide storage space, folders to save search strategies and specific citations, and e-mail alerts for new citations. In PubMed this feature is called MyNCBI. After a short registration process that requires you to create an account with an ID and password, MyNCBI allows you to save a search strategy that can then be rerun at a later date. Search strategies can also be rerun to display only the new citations added to the database since that last time the strategy was run. Selected individual citations from searches can be saved temporarily to the ClipBoard or permanently saved to the Collections folder in MyNCBI. Both the Clipboard and Collections eliminate duplicates and collect the citations into one set, which can then be exported to EndNote or other bibliographic management software. CINAHL has a similar feature called MyEBSCOHost, which provides folders for keeping track of search strategies and selected citations.

Reference Management Software

Reference management software allows you to download and organize references from the major databases, create bibliographies in customized formats, and easily incorporate references into working manuscripts. The references can be entered manually or imported directly from databases, such as PubMed and CINAHL. You should check with your medical library to find out which programs might be supported at your institution. In addition, there are a growing number of Web sites that provide free online reference management tools.

KEEPING CURRENT WITH NEW EVIDENCE

The amount of new information added to databases and the Web is staggering and can present a challenge to health professionals who need to keep current on specific research topics and their own professional literature. As an example, 2000 to 4000 completed references are added each working day to PubMed. Fortunately, many of the databases provide automated current awareness services that allow professionals to easily review new research.

PubMed through its MyNCBI service allows you to generate an e-mail alert whenever new citations are added to the database that match a specific search strategy. When a citation is added to the database, an e-mail message is sent with the abstract. This alert service can also be generated for table of contents of current issues of journals indexed in the database. (See http://ncbi.nlm.nih.gov/sites/myncbi/about/ for more information.) MyEBSCOHost from CINAHL offers a similar service. Many individual journals also offer an automated table of contents service; however, most require an individual subscription to activate their service.

In 2011, the American Physical Therapy Association unveiled PTNow (http://apta.org/PTNow/), a Web-based portal designed to give physical therapists easy access to various types of evidence. It allows users to find original, assessed, and synthesized evidence for both background and foreground questions; to search related Web sites; to ask questions of experts; to access information on diagnostic, prognostic, and outcome tools; and to download patient education materials.

Blogs, Wikis, and RSS Feeds

The Web has evolved from a static, view-only platform to an interactive and engaging tool for exchanging ideas and conversing about issues. This evolution, sometimes referred to as Web 2.0, utilizes social networking tools to encourage communication. Some of these tools, specifically blogs and wikis, are being used to share news, information, and opinions and to create a greater sense of community among people with common interests. A blog is a Web site, usually authored by an individual, that offers commentary and news on a particular subject. The content can include text, images, links to other Web pages and multimedia, and the option for readers to leave comments in an interactive format. Posts are updated frequently and presented in chronological order with links back to older material. One of the great appeals of blogs is that they use very simple editing screens that do not require any experience with HTML or Web-authoring tools. This may help explain the estimates of over 50 million blogs in existence. Most of these are authored by individuals and cover a very wide array of topics from the very personal journals to social and professional issues to important medical information. Blogs can be very useful sources of current information, but like any other resource from the Internet, you must evaluate the blog for authorship, potential bias, and authoritativeness before you accept and use the information presented.

A wiki is a collaborative Web site that allows a group of people to develop and edit content on a topic. One of the most well known wikis is Wikipedia (en.wikipedia.org), the largest, free, online encyclopedia currently available on the Internet. Like blogs, the quality and value of the content of wikis can vary dramatically, and it is important to always read the content with a critical eye. However, wikis tend to focus more on providing content and information, rather than opinions and personal accounts.

RSS (Really Simple Syndication) is an easy way to manage all the new content that is being published daily on the Internet. The RSS feed provides a mechanism for readers to receive current, unread content from blogs, wikis, and other Web sites at a central point or Web page, called a "feed reader." This eliminates having to go out and visit each blog, wiki, or Web site to read what has been newly published. The feed reader can organize the selected feeds by subject area and displays the newest unread postings on a single Web page. RSS is also being used by PubMed and CINAHL to allow searchers to generate an RSS feed of a specific search strategy to display the new results in a feed reader (rather than e-mail), with links back to the citations in the database. Example of RSS feed readers are Google Reader (www.google.com/reader/) and Bloglines (www.bloglines.com/).

MOBILE DEVICES

Over the last several years smart phones and tablets have become common devices for accessing information resources. Many of them are Web-based, so users can access Internet resources just as they would do from desktop computers. In response to this growing use, libraries and resource producers are reformatting their pages and products to conform to the size of a smart phone or tablet screen. A second way that these devices can access resources is through specific programs or applications (often referred to as "apps"). These applications are usually self-contained and downloaded to the device so that they can function without being connected to the Internet. There are 100s of medical applications available for a variety of operating systems, including Apple, RIM, Android, and Symbian.

Applications can be grouped around three activities: patient education, reference, and clinical tools. Check your local medical librarian, iTunes Apps Store, or Android Market for other applications. Blogs and wikis are also excellent sources for reviewing and ranking medical applications.⁶

CONCLUSION

The practice of physical therapy requires finding current relevant evidence to answer clinical questions about the care and management of patients. Formulating a well-focused question and becoming familiar with the resources available to you can make finding information easier and more productive. The next step, once you have found relevant studies or information on your question, is to evaluate the evidence for validity and review the results for applicability.

SELF-ASSESSMENT

1. Using one or two of the questions you developed in Chapter 10 about Mr. Ketterman's care, use the filtered resources discussed in this chapter to find answers to your questions.

2. Now try to find answers to the same questions using unfiltered resources.

3. Compare the types of evidence that you find using both of these search approaches.

4. If you are unable to find any answers in half an hour, reach out to reference or medical librarians or colleagues to get help.

CONTINUED LEARNING

• Visit PTNow and some of the blogs or wikis discussed in this chapter. Prepare a presentation on the resources you find there for your colleagues.

• Use some of the tutorials identified in the chapter and then present these search options to your colleagues as ways for them to find information to support their practices.

• Visit a few of the myriad resources available for patients (Web sites, blogs, wikis, apps, etc.) and determine which ones and why you will recommend them to your patients.

REFERENCES

INTRODUCTION

Having found evidence that relates most closely to a clinical question, the next step in the evidence based practice (EBP) process requires the careful reading and critiquing of the evidence. But this is perhaps the most daunting task facing clinicians who wish to practice in an evidence based manner—determining whether evidence found in the literature can be of value to their patients. As the sophistication of physical therapy research increases, the task of evaluating any research report for its accuracy and truthfulness becomes more challenging. After all, the evidence can only be of value to your patient if it is derived from a valid study. Validity, used here to mean trustworthiness, is an essential characteristic of any individual study. The more the results of a study can be trusted, the greater its validity in helping to make an informed clinical decision. An understanding of some of the methodological and analytical standards for conducting valid, high-quality clinical research experiments is a good starting point to build one's critical appraisal abilities. In this chapter we will focus on understanding the elements of good research. Chapters 13, 14, 15, 16 will then apply the principles of good research practice to the various types of evidence, ranging from single studies to systems (Fig. 12-1).¹

It is not the intention of this text to provide direction for researchers, nor to be a full source of all the details of research design and management that a clinician might find interesting. Rather, we have attempted to identify the essential elements that should be considered in assessing the literature. Because there are many such elements and you may wish to focus on them specifically when reviewing articles, we provide an overview of what will be covered in this chapter (Box 12.1).

RESEARCH DESIGN

We will compare two large categories of research designs, experimental research designs and observational research designs. The research design is selected by the investigator based on the research question of interest. As you have seen in Chapter 10, questions may relate to selecting the best examinations in order to determine a diagnosis for the patient or may relate to the risks and benefits of a particular intervention. Experimental research designs provide the strongest evidence for causal relationships between an intervention and an outcome of care. They are the designs typically used to demonstrate the efficacy of an intervention. efficacy is a measure of the capacity of an intervention to actually produce a change. Observational research designs can also provide good evidence for intervention studies and are more frequently used to study diagnostic or prognostic questions. They are designs typically used to demonstrate the effectiveness of an intervention. effectiveness is a measure of the ability of the intervention to bring about the expected change in the real world.

Research designs in biopsychosocial sciences stem from one of two philosophical paradigms, the quantitative paradigm or the qualitative paradigm. The difference between these two approaches as to what is knowable is described in Chapter 8, and the qualitative research paradigm and research designs are discussed there, including criteria with which you might determine the value of a study that uses a qualitative research design. The research designs described in this chapter stem from the positivist philosophy, which purports that truth is independent of the investigator and knowable through direct and carefully controlled observations, interventions, and measurements of subjects. There are many rubrics that categorize research designs in medical science, with discrete definitions and methodological characteristics. In this text we will describe several of the designs most commonly found in literature useful to physical therapists and identify the questions that are useful for determining the value of findings derived from each.

Experimental Research Designs

Two categories of quantitative research designs are useful for organizing one's thoughts (Fig. 12-2). The first category is experimental research, within which we will discuss the type of experimental research that epitomizes this category, the randomized controlled trial (RCT). An experimental research design is selected when the investigator wishes to examine if a cause-and-effect relationship exists between one or more interventions and the subsequent outcomes. Traditional criteria for true experimental research designs include random selection or allocation of research subjects to groups, the inclusion of a control group for comparison to a treatment group, and purposeful manipulation (dose, frequency, or duration) of the treatment variable. Of these three criteria, randomization of subjects is often considered the single most important element. Randomization of subjects to treatment groups helps ensure that the groups will be equivalent on important measures at the beginning of the study, thus giving the best measure of the relative effects of the two contrasted treatments. These research designs also employ tightly controlled research methods in order to provide the strongest possible statement about observed differences in groups at the end of the experiment.

In addition to true experimental research designs, some researchers may use a variety of designs that are similar to experimental research but do not quite meet all the rigorous requirements of true experimental research. In the past, these designs might have been referred to as quasi-experimental. Quasi-experimental designs share the purpose of establishing a cause-and-effect relationship between the treatment and the outcome, but fall short of the design standards of true experimental research, perhaps because it is unfeasible to randomly assign subjects or to use a control group.

Many aspects of good experimental research design are difficult to accomplish in physical therapy research, and more recently, some have used the term, practical clinical trials (PCTs) to indicate a research design that is not quite as rigorous as that found in a RCT.² Calif recommends such designs, commenting in an Institute of Medicine Roundtable on Evidence-based Medicine, "the sheer volume of clinical decisions made in the absence of support from randomized controlled trials requires that we understand the best alternative methods when the classical RCTs are unavailable, impractical, or inapplicable."² Investigators have called for clinical trials that, although perhaps somewhat less rigorous, will better meet the needs of the clinician and the patient by including a comparative or alternate therapy that is relevant to the choices of the patient.^3,4,5 Whether these PCTs are considered to be experimental or non-experimental research is not as important as making a clear judgment about the value of each study to you and your patient.

Observational Research Designs

The second category of research designs is termed observational designs, or nonexperimental designs. While within this category there are many, quite different designs, each of them shares one characteristic: they are conducted by making careful observations of clinical practice as it happens, or happened, rather than planning a controlled test or manipulation of treatment. Patient care goes on as planned, and close observations are made about relationships among variables, treatments, and outcomes. These designs are used when the knowledge to posit testable hypotheses for experimental research is unavailable; when the phenomenon of interest is too complex to be assessed with a clinical trial; or when the purpose of the study is not related to interventions, but to the quality of the tests or outcomes of care. Observational research designs are quite varied; some are most useful in answering questions about interventions, although lacking the strength of the RCT, and some are best suited to questions about diagnostic accuracy of tests or the natural history of an illness.

Brief definitions of the major observational research designs follow:

• Cohort: In this design, the researcher is typically interested in describing the future health outcomes in one cohort of subjects who have an exposure to a risk factor or will be given a certain intervention, and one cohort that is similar, but did not have either the exposure or intervention. Both groups will be followed forward in time and measured frequently to determine the occurrence of the outcome in each group.

• Case-Control: In this design, the researcher will identify two groups of subjects: the case subjects who have an outcome of interest, for example, a disease; and a group of subjects similar to them who do not have the outcome. Then the research uses medical records or interviews to look back in time to determine when and how each group member was exposed to the suspected causative factor.

• Cross-Sectional: With this design the researcher will identify a group of appropriate subjects and will measure their outcomes and their exposure to potential causative factors at the same time, generally just one point in time. This design is also typically used in studies of the diagnostic accuracy of tests, as the subjects are given two or more tests at one point in time.

• Methodological: These designs include repeated measures on subjects with the outcome of interest, for example, lax knee ligaments, in order to assess the reliability or validity of tests.

• Descriptive: These designs sample specific groups of subject, often using random sampling, and measure opinions or collect data about the course of an illness. These designs are useful for investigating satisfaction with health care.

• Case Study: This research design calls for careful measurement and description of typical clinical practice and is useful for documenting care given to patients with unique circumstances or responses to treatment.

DETERMINING VALUE IN STUDIES OF CLINICAL INTERVENTION

Our framework for assessing value has two components: research validity (sometimes termed experimental validity) and statistical conclusion validity. Traditional research methodologists Campbell and Stanley⁵ provided one of the first frameworks for evaluating the quality of experimental research by describing the elements of research necessary to establish a causal inference. They proposed that an experiment designed to demonstrate a cause-and-effect relationship between interventions and outcomes should be judged on a set of criteria that define the research validity, or truthfulness, of the study. Research validity has two components, internal and external. We will focus primarily on internal research validity, which are those research procedures that create the greatest trust in the results of the study. For experimental studies this means having trust in the inference of the causal relationship demonstrated in the study. For observational studies, this means having trust that the phenomena being observed are accurately represented. In both designs, research validity helps to determine the extent to which bias has been controlled in the study. Bias, in a research context, refers to the tendency toward systematic errors that can arise from the design, sampling, and measurement used. Therefore, we are interested in criteria that allow us to assess how well the researchers have controlled for alternative explanations for the outcomes of the study, other than the influence of the variables of interest. These criteria are focused on the methods of the study and they will influence our confidence in its outcomes.

External research validity concerns how generalizable the findings are to patients who were not in the study. Commonly, external validity is concerned with asking how similar are the people in the study and the circumstances (time and place) of the study to our specific patients and the circumstances of the care we plan to give.

Another set of criteria developed by Campbell and Stanley⁶ helps us evaluate the appropriateness of the statistical analysis of the study, or the statistical conclusion validity. Critiquing the internal and external research validity and the statistical conclusion validity will allow us to adequately discuss the value of the research reported in studies that appear to relate to our clinical questions (Fig. 12-3).

Criteria for Evaluating Internal Research Validity

The information needed to assess internal research validity is generally found in the methods section of a paper. This section contains very detailed descriptions of the procedures, measures, instruments, subjects, setting, timing, and participation required of the subjects. In experimental designs, these aspects of the study design are controlled very specifically so as to have the best chance of identifying a cause-and-effect relationship between the intervention being tested, also called the independent variable, and the outcomes measured, or the dependent variables. As you read the methods section of a randomized controlled trial (RCT), you will be judging whether or not the authors exerted sufficient control to eliminate bias in the procedures, measures, sample selection, or any other important aspect of the study. Keep in mind that most authors find that the assessment of truthfulness in research is a subjective evaluation, one that places the study on a continuum from strong to weak.^7,8 We will look at three categories of good research practice:

1. selection of the correct research design,

2. selection and treatment of subjects, and

3. selection and methods of intervention and measurement of outcomes.

Did the Research Design Maximize the Prevention of Bias?

Each of these types of research designs, experimental and observational, brings strengths and weaknesses to the study of best physical therapy practice. Recent studies of the impact of research design on the evidence for practice have shown no difference in results when observational studies are compared to RCTs,⁹ while others argue that this is not the case. Califf² provides two examples of incorrect conclusions drawn from multiple observational studies in cardiology, identified as incorrect practice only after sufficient RCTs were performed. He comments on the hope that analytical methods might enhance the causal relationships in observational studies, "… no amount of statistical analysis can substitute for randomization in ensuring internal validity when comparing alternative approaches to diagnosis or treatment."² The evidence to support clinical decisions in physical therapy research is comprised of both experimental and observational research designs, with some appearing less frequently than others (cohort and case-control). Table 12.1 provides a summary of examples of research designs you will find in the literature for questions about interventions, diagnosis, and prognosis in physical therapy.

Did the Selection and Treatment of Subjects Avoid the Introduction of Bias?

People who consent to participate in health care research studies may not be patients of any practitioners at the time of the study, because they do not require health care. Alternatively, they may be patients under the care of the investigator or another practitioner who has informed them about the study. In both cases, these people are referred to as subjects in terms of their role in a research study. This distinction is very important to the investigators conducting the study and to the clinician reading the study. The investigator must distinguish between procedures given to the patient that are tightly controlled research procedures and others that are customary care. Readers of research are interested in learning sufficient detail about the study subjects so that they can determine if the subjects in the study are sufficiently like their own patients; otherwise, the study might be of less value to their practice.

The selection and treatment of the subjects who participate in research studies can be a significant source of bias. A myriad of criteria might be used to determine if just the right study subjects were included and if all the proper procedures were used with them, including inclusion and exclusion criteria that will provide the best sample for the study, the proper allocation of subjects to study groups, the importance of tracking study subjects across time, and controlling what the study subjects know during the study (Fig. 12-4).^10,11,12

Obtaining the Best Sample of Subjects for the Study

One of the hallmarks of experimental research in the social sciences is the selection of a random sample of subjects, but this process is often unavailable to or cost-prohibitive for health care researchers. Instead, much of the research is conducted with samples of available subjects, called samples of convenience. Once a research question and an appropriate research design are selected, the next decision is, who should be in this study? The resources to study everyone to whom the research question applies, often referred to as the population of interest, would be unavailable in most health care research. The method of selecting a smaller group of patients to whom the research question applies is referred to as sampling the population, and this is undertaken with the hope of achieving a manageable group of research subjects who appropriately represent the original population. This group is called the sample, and good research practice requires the clear definition of who can and cannot be in the sample so as to maximize the research design's purpose to discover a causal relationship between intervention and outcome, if one exists, while not making the sample so narrowly focused that it no longer represents the population of interest. When this happens, clinicians find the research to be less useful to their own practice.

The use of criteria that a person must have to be eligible for the study, inclusion criteria, and those that they must not have to be eligible for the study, exclusion criteria, allows the researcher to recruit people to become research subjects. Some research designs may require very intricate inclusion and exclusion criteria for subsamples of the study, but we will use the structure of an RCT with an experimental and a comparison group to illustrate good sampling procedures. The first step to critique is the method used to recruit all subjects. Ask yourself if the researcher used sampling criteria that served to eliminate bias in the subjects chosen, for example, by disqualifying subjects who had multiple comorbidities to the one being studied. Having such subjects in the study could introduce competing explanations for the results. The study should provide a description of objective characteristics of the sample selected so that you might judge how similar or homogeneous a group was identified; for example, were the subjects similar in the severity of their injury or illness, the length of time since onset, and the occurrence of surgery or not? It is important that the overall sample of subjects is homogenous on important clinical, demographic, and temporal characteristics, not only to decrease sample bias but to help ensure equivalent groups once group allocation is undertaken.

Proper Assignment of Subjects to Groups

Many of the research designs described in the previous section make use of groups of subjects who receive different treatments or different levels of treatment in the study. If a comparison of X with Y is planned in the RCT, the sample will need to be assigned to one or the other group. Because the sample has generally been created using available subjects, it is crucial that the significant benefits of randomization be applied to research designs at this point. While sampling is generally not done randomly, random assignment, or allocation of subjects to groups, provides a vital element of experimental validity to the research process. Randomization helps distribute subjects with both known and unknown characteristics equally into all study groups, lending confidence that the groups were made as equivalent as possible at the beginning of the study. If the two groups are different before the study starts on some characteristic related to the intervention or the outcome, then bias has been introduced and weakened the ability to identify a cause-and-effect relationship. If one group had a very different prognosis for recovery than the other, imagine how that would influence your thinking about the outcome of the study. Again, the study should provide an objective description of the equivalency of the groups, using tables or charts, and often a statistical test, as evidence of the effectiveness of the randomization procedure.

There are several methods to accomplish random assignment of subjects to research groups, most of which depend on whether or not the whole sample is available for randomization at the start of the study, or if randomization must be done consecutively as subjects are enrolled. This is the case for most studies that use a sample of convenience. In this case, an ordered list of subject identification numbers is randomized and subjects are assigned to groups as they enter the study. Clinical trials often use another assignment technique called matching assignment, which allows the investigator to exert some control over the equivalency of the groups in addition to that provided by randomization. In matching assignment, each subject is categorized on an important and possibly confounding variable, for example sex or leg length, matched with another subject with similar characteristics, and then randomly assigned to a group, such that there are similar numbers of women in each group or equal numbers of subjects with short leg lengths in each group. In this way, the researcher does not leave it to chance (the randomization process) to equally distribute subjects with known attributes that might introduce bias into the study, should they by chance all end up assigned to one group.

We will discuss later in the chapter the importance of keeping clinicians who are involved in treating or measuring research subjects unaware, or blind, to the group assignment of the subjects. However, it is also considered important to know whether or not the person enrolling subjects into the study is aware of the group to which the next subject enrolled will be assigned. Good research practice will keep this consecutive order assignment concealed from the person enrolling subjects to avoid a bias introduced by the clinician who may not want a potential subject to be assigned to the control group. This practice is called allocation concealment.

Tracking Subjects Through the Study

Bias can be introduced into a study if the researcher loses track of subjects who do not complete their participation. If the study takes place over days, weeks, or months, it is likely that some study participants will become ill, reinjured, discouraged, or in some way no longer willing or eligible to participate in the study. Other participants may recover sooner than anticipated and no longer wish to participate in the study. Still others stop participating and the researcher never learns why. It is important to account for each subject at the end of the study, and particularly so if these dropouts have happened with differential frequency among the groups. At the end of the study, researchers should remain confident that the randomization performed so carefully at the beginning is not negatively affected by differential rates of attrition in the groups. Standard schematics or flow diagrams are typically provided in a study to document the progress of participants through each stage of a clinical trial (Fig. 12-5).^13,14

Controlling What Study Subjects Know

Controlling who knows what, when, in an experimental study is an important tool for avoiding various types of biases introduced by those involved. Here we discuss controlling what the subjects know specifically about their involvement. Good research practice requires that all potential and enrolled subjects be fully informed as to the procedures that will be used in the study so that they might make the best judgment about their involvement. In some studies, a subject will learn from the informed consent process that they have a one in three chance to be in a group that gets exercise A, exercise B, or no exercise at all. The subject is told in the informed consent document whether or not they will know to which group they have been randomly assigned and how that randomization will take place. In studies of medications, it is easier to keep subjects blind to their group assignment than it is in studies of procedures, surgeries, or physical therapy interventions. Thus the ability to include this good research practice is limited in rehabilitation research. When the subject cannot be blinded from the group assignment, then care is taken to utilize outcome measures that may be less influenced by the subject's knowledge of his or her group assignment, for example, physiological measures of body temperature, wound healing, and range of motion; however, it is not certain that these measures could be influenced by the subject themselves.

Did the Selection and Methods of Interventions and Measurement of Outcomes Avoid the Introduction of Bias?

To answer a question about which intervention may have a better effect on health outcomes, the researcher must make many choices for the conduct of the study. Methodological decisions focus on two major components: the study procedures and the study measurements. Good research practice in these areas must blend a keen focus on the research question with an eye to avoiding bias in all decisions about procedures and measures. For example, the most intricately designed study procedures could be wasted if the selected outcome measures are not the best or are performed in a biased manner. We will look first at the decisions to be made about the study procedures regarding interventions.

Selection of Interventions

A researcher wishes to investigate whether eccentric or concentric quadriceps strengthening exercises will produce better outcomes for patients who received a total knee arthroplasty (TKA). It is obvious to the researcher that either type of exercise will be better than no exercise, so there is no need (and perhaps an ethical concern) for having a control group that receives no exercise. The researcher knows one of the study variables will be type of exercise, and only two types, eccentric and concentric exercise, will be studied. What about the dose of the exercise for each type? Should subjects perform the intervention exercises daily or less frequently? How many repetitions of each exercise? Should subjects be required to visit an outpatient physical therapy clinic or perform the intervention as a home program? How many days or visits should be expected until an effect can be found? Will exercise equipment be involved? Will therapists be required to deliver the intervention directly? If so, what bias might that introduce? It is very difficult to keep the treating physical therapists in clinical trials unaware of (blind to) the group assignment of the subjects, but good research practice requires that the researcher consider whether this can occur in the study, to avoid the bias introduced by a research team member who might favor one group outcome over another.

Measurement of Outcomes

The second set of methodological decisions concerns the measurement of the outcomes of the study. In the study of choice of exercises for patients who have undergone TKA, the investigator must decide what outcomes to choose in order to determine which exercises are better for the patients. Over the years, investigators of physical therapy clinical trials have expanded the types and numbers of outcome measures considerably. Today, a clinical trial is likely to have a mix of outcome variables, often classified as primary and secondary, based on the importance to the investigators and to patients. The measures may be selected to assess physical and social well-being, as well as outcomes that address the processes of care, such as patient satisfaction. A wide range of measures can be selected that represent elements of the WHO International classification of Functioning, Disability, and Health (ICF) model.^15,16 Applying this model, described in Chapter 1, to the selection of outcome measures could result in assessing body structures and functions with, for example, ROM or strength about the knee and a 6-minute walk test and assessing activities and participation with, for example, the Knee injury and Osteoarthritis Outcome Score (KOOS),¹⁷ The Guide to Physical Therapist Practice includes a compendium of over 800 tests and measures from which to select the measures used in a study.¹⁸

With so many available measures, what criteria might be used to select the best ones for a specific study? Two fundamental principles of measurement are the reliability and validity of the measure.¹⁹ These characteristics of measurement provide the clinician and researcher with confidence that they can rely on scores derived from these tests to be dependable (reliable) and accurate (valid). It is important to remember that no test or measurement process is ever perfect, and errors of one type or another are associated with every outcome measure. This is particularly true for measures that require the interaction of the therapist with the patient, as compared to tests that the patient completes independently, like a survey. Reliable measures are ones for which repeatable scores would be achieved when the test is administered more than once, over a time period in which it can be reasonably expected that the phenomenon of interest does not change, for example, two measures of isometric quadriceps force production measured on a dynamometer, taken 30 minutes apart. Valid measures are said to be accurate, and by that we mean they measure what we intend for them to measure, and not another phenomenon. If we believe that an isometric quadriceps force production test on a dynamometer is an accurate measure of the concept of quadriceps strength, then the measure can be said to be valid for the purpose of the study. When there are several tests that purport to measure the same concept, studies are performed for the purpose of comparing one test to another and thereby establishing a description of the validity of one test in comparison to another. We will expect the author of the clinical trial to report the reliability and validity of the primary outcome measures in the study as a means of assuring us that the best tests for the study were selected.

Two other important decisions should be evaluated when considering the methodological aspects of measurement in a study: the timing of the measurements and who takes them. In general, the closer in time measures are taken to the phenomenon of interest, the less error will be represented in the score. In designing the study of eccentric versus concentric quadriceps strengthening exercises over a 12-week period, the researcher must decide when to take the measurements. The two obvious time periods to be selected are before the study starts and when it ends. If the pretest measure is taken too long ahead of time, an unexpected occurrence might cause the subject to become weaker before the study starts, thus introducing error into that subject's score. Good research practice will include measures closely timed to specified points in the study for all subjects. One other aspect to consider in terms of the usability of the study to your practice is whether enough measures were taken, especially follow-up measures, ones that typically follow the conclusion of the study. Follow-up measures allow assessment of the long-term effects of interventions, so that clinicians might learn how to best time interventions over an episode of care for their patients.

The most important methodological decision related to outcome measures is whether or not the person taking the measures is blinded to the group assignment of the subjects. If the clinician giving the intervention cannot be blinded, it is very important that a different person take the measures. Even this amount of blinding of the measurer can be difficult in physical therapy studies, since interventions often change the appearance of a subject's physical body. In such studies, the timing of the measurements may need to be adjusted, or instructions given to subjects not to reveal his or her group assignment, as well as other safeguards put in place to maximize the objectivity of the clinician taking the measurements.

Evaluating External Validity

One additional aspect of good research practice that you might consider in determining the value of a study to your practice is how generalizable the results are to your practice. This is referred to as the external validity of the study. External validity asks, can we apply the results from a sample of patients in the controlled environment of the experiment to real life? If the controlled aspects of the setting, the subjects, and the timing of the study design are similar to your practice environment and patients, then the study findings will be generalizable to your patients. If the methods of the study bear little resemblance to how you might carry out the intervention in your practice, then you cannot be as confident that you will find the same outcomes as reported in the study. Generally, these are questions a clinician must answer based on his or her best clinical judgment. The desire to improve internal validity can sometimes result in less external validity; the reverse is also true.

Evaluating Statistical Conclusion Validity

Statistical conclusion validity refers to the degree to which the analysis performed on the data in the study allows you to make the correct decision regarding the truth or approximate truth of the hypotheses tested. This is different from the process we just reviewed, the goal of which was to determine if we could trust the causal relationship between the intervention and the outcomes of the study, based on the methods of discovery used in the study, or its experimental validity. For most clinicians, research validity is the easier conclusion to draw about a study. For some experimental studies, the procedures described have clearly introduced some type of bias, for example, failing to mask the examiners from the subjects' group assignment. It is then both easier and logical that the reader will consider research validity before reading the data analysis and results section of the paper. With practice, the reader will begin reading the results section with some expectations in mind about the outcomes of the study. For example, you might anticipate small between-group differences because the intensity of exercise given to the experimental group was, in your clinical judgment, insufficient to create a large effect. It is best to be aware of your propensity to accept the results before you begin reading them, for the statistical analysis may indicate that a treatment was beneficial to one group of subjects as compared to another. Such statistical differences cannot be considered truthful unless you are satisfied with the experimental validity of the work.

If you are satisfied with the research validity of the study, the second component to determining the usefulness of this study to your practice is to evaluate its statistical conclusion validity. Domholdt defines this simply as your assessment as to whether or not statistical tools have been used correctly to analyze the data in the study.²⁰ In the following section we will present an overview of basic statistical principles and terms and define their use in analyzing the data collected in experimental and observational studies. To understand statistical conclusion validity, one must master a certain amount of measurement and statistical theory.

Norman and Streiner suggest that the main purpose for understanding statistics is to be able to distinguish true differences from natural variation.^21,p2 This is what we wish to know when reading in the results of a study that, for example, a 5-point difference on a pain scale existed at the end of a comparison between a group receiving an experimental treatment and a group receiving a standard treatment. Were these two groups truly different from each other at the end of the study, or does this difference reflect natural variation in subjects' pain response to treatment? We might also say that the difference of 5 points between the groups might have happened by chance, and this is another way to describe natural variation between and within individual subjects. Without statistics to help us make sense of this statement: "there was a 5-point difference in pain scores between groups at the end of the study," we would not be able to have confidence that this was a true difference, caused by the treatment.

There are two large families of statistics that you will find in most studies: descriptive statistics and inferential statistics. Descriptive statistics help define the average subject, intervention, and outcomes, as well as the variability in these data. They are important analyses to present to the reader first, in order to describe the subjects, the interventions, and the outcome measures. The subjects selected for a study are referred to as the sample, and the numerical measures of central tendency and dispersion calculated for the sample are called statistics. These calculated statistics are used to project or to infer what might happen should all possible appropriate individuals be studied. The term used for all possible appropriate individuals is the population. Thus, we conduct studies on selected samples of subjects in order to infer what might happen could we study the entire population. The numerical measures of central tendency and the dispersion hypothesized for a population are called parameters. The second large family of statistics is then termed inferential statistics, for they allow inferences from the sample statistics collected to the population parameters, which are generally unavailable. One can think of this as trying to discover a causal relationship in a sample from one study, which can then be recommended with confidence to the entire community.

Descriptive Statistics and Point Estimates

Data are collected to describe subjects following rules of measurement that allow consistent communication of the value of what is being measured. Data can be communicated, or measured, as names, numerals, or numbers.^20,21 It is important to distinguish numerals from numbers, as numerals generally cannot be entered into mathematical formulas because they are labels and do not have quantitative value (Table 12.2). There are four classical categories, or levels, of measurement referred to in the literature: nominal, ordinal, interval, and ratio. The distinguishing elements of these categories are the mathematical operations one can perform upon them, based on the units of the measurement and the presence or absence of an absolute zero point.

The three types of measures (name, numerals, and numbers) can also be classified into two broad categories that tell us something about the form they may take: discrete and continuous measures. Discrete measures are those that can only assume a limited set of values (generally, a known set of values). Data measured with names and numerals are generally discrete measures. Data measured with numbers can be either discrete measures (whole numbers only, for example, number of pregnancies a woman has had) or continuous measures (those that can be represented with decimals, for example, timed up-and-go score of 13.5 seconds). These distinctions about measurement of data influence later data analysis and may influence your judgment about the precision of measurement used in a study.

One of the first tables one might find in an experimental or observational study contains descriptive data about the subjects as a total group and/or divided into their assigned groups. When you look at these tables, you often find the data for several variables provided, with specific statistics for each type of data. When more than 10 subjects are in a study, it is useful to calculate statistics that tell something about the distribution of all the scores, because there is too much information to use otherwise. (For small studies of 10 or fewer subjects you might find a table that lists each subject and their individual scores). These statistics are designed to help picture the distribution of all the scores for each variable. The descriptive statistics typically found in these tables are defined in Table 12.3. An example of such a table is found in Table 12.4.

Measures of central tendency are single values that tell you something about the mid-point in a distribution of numbers. Measures of dispersion are designed to tell you something about how variable the scores are around the measure of central tendency. The age of subjects is often reported in studies as a mean and a standard deviation; for example, in Table 12.4 the mean age of the experimental group subjects was 37 years with a standard deviation of 6.5 years. Instead of the mean, the median or the mode may be used to describe the center of the distribution if the data are discrete or if the continuous data have extreme scores, such that the mean might be misleading to the reader. In Table 12.4, the median and range are used to describe the distribution for the variable "weeks since onset of pain," since at least one person in the experimental group had an extreme score of 15 weeks. Together, the measures of central tendency and dispersion give the reader an idea of how the distribution of all the scores for subjects in that group would look. The measures of dispersion are also selected to fit the type of measure being reported, so understanding what each means can give the reader an understanding of how spread out the scores are for that measure. For example, compare the two means and standard deviations for the variable doses of pain medication taken in the past week in Table 12.4. The two groups have similar means, but the variability in the experimental group is twice that in the control group.

These measures of central tendency and dispersion are calculated from one sample, and it is quite possible that another study reported in the literature measuring the same variable may report quite different statistics. Which descriptive data are the most accurate to use in making inferences to the population of interest? If study A reports a 5-point mean decrease on a pain scale following an experimental intervention and study B conducted on the same intervention reports a 7-point mean drop in pain scores, which can you expect to happen with your patients if you use this intervention? There are additional statistics that help enhance predictions of measures of central tendency such as means as well as inferential statistical estimates, discussed later.

The standard error of the mean (SEM) is a useful statistic to understand how stable inferences about a population mean are, and this statistic is highly influenced by the sample size. Examining the calculation of the SEM (Table 12.5), one can see that large samples will have a smaller SEM, giving more confidence in the prediction of the population mean, while small samples will have larger errors in estimating the mean of the population. The SEM is a component of the formula for another statistic, the confidence interval (CI), which helps understand the accuracy of point estimates like the mean. This statistic is conveniently named, as it gives a range of scores within which the true value for the mean is expected to be found. The data in Box 12.2 illustrate how useful confidence intervals are to interpreting descriptive data.

Assuming that both studies enrolled 25 subjects and study B had both a higher mean and standard deviation, you can see that the range of the CIs for the two studies also differ similarly. From study A, we are 90% certain that if the true population mean is not 5, as calculated in the study, then it falls between the values 4.6 to 5.4. From study B, we are 90% certain that if the true population mean is not 7, as calculated in the study, it then falls between 6 and 8. The CI for study A is smaller than that for study B because the standard deviation around the mean found in study A is smaller. So, if you wish to know how much reduction in pain scores you might expect from this intervention, you can make use of both CIs to predict that the change to expect in your patients would be no smaller than 4.6 points and might be as high as 8 points.

Homogeneous and Heterogeneous Data

Two useful terms to understand when evaluating either a total sample of a study or the two or more groups in a sample, are homogeneous and heterogeneous. If subjects in a group have very similar scores on the measures taken, you can say the group is homogeneous in, for example, the amount of drugs they are taking during the study. If you examine the means and standard deviation/variance for a control and experimental group and find the standard deviations to be quite different, you can say the group with the larger standard deviation is more heterogeneous than the other group. Homogeneous groups are those that are composed of subjects who score similarly on the measures, and heterogeneous groups have subjects who score quite differently from each other. The statistical analysis that is performed on the measures to look for statistical differences between groups is influenced by the homogeneity of the groups in relation to each other. If the control group is quite similar in scores (for example, most subjects changed less than 5 points from pre-test to post-test) while the experimental group is extremely varied in scores (some subjects changed only 2 points and some subjects changed 15 points and everything in between), then an important assumption for the statistical analysis, homogeneity of variance, is violated and the analysis may be incorrect. Random allocation of subjects to their groups in RCTs is very important to help achieve two or more groups who have homogeneity of variance. Inspecting the descriptive statistics for each group in the study design will allow you to decide if the groups are homogeneous.

Normal Distribution

The normal curve, or bell-shaped curve, is a concept important to understanding the measures of central tendency and dispersion we have just defined. When large numbers of scores are analyzed, a typical bell-shaped distribution for that score emerges that can be defined mathematically in order to predict the probability of any one score occurring. This is very helpful to clinicians in selecting what we consider to be normal values for clinical tests, for example, defining normal systolic blood pressure. The normal distribution provides us with a mean for systolic blood pressure and also an understanding of the range of scores one could expect in patients. Particularly high or low scores, which may not be normal and could require treatment, are easier to identify.

In normally distributed data:

1. the mean, median, and mode will hold the same value, so any of these statistics can be used to describe the middle of the distribution;

2. the probability of any one score falling within the middle of the distribution, defined as the mean minus 1 standard deviation or plus 1 standard deviation, is 68%; thus the scores that cluster closely around the mean are the most frequently occurring scores;

3. the probability of any one score falling within the area of the distribution, defined as the mean ±2 standard deviations, is 95%;

4. the probability of any one score falling within the area of the distribution, defined as the mean ±3 standard deviations, is 99%; capturing almost all scores. Since only 1% of scores fall more than 3 standard deviations above or below the mean, we can understand these scores to be very rarely occurring scores.

Applying these assumptions to any presentation of a mean and standard deviation given in a table allows us to understand how widely dispersed all likely scores are. Scores that are unexpectedly high or low could represent errors of measurement or could be recognized as rare values. Many of the inferential statistical tests that are used to tell us if statistically significant relationships or differences exist between variables or groups have as an assumption that the scores are normally distributed.

Visual Representations of Measures of central Tendency and Dispersion

A graphic figure frequently used to express the measures of central tendency and dispersion for a set of scores is called a box plot. Figure 12-6 shows the conventions used to construct a box plot. The lower and upper margins of the box are the score values occurring at the 75th percentile of the distribution (upper quartile) and the 25th percentile of the distribution (lower quartile), so the box itself illustrates what is called the interquartile range, or the middle 50% of the distribution. If the box demonstrates a wide variation in scores from the upper quartile to the lower quartile, it will be quite wide; a distribution with more closely clustered scores will be narrower. A symbol (* or +) or a line is often used within the box to indicate the median of the scores. If the median falls equally between the upper quartile and lower quartile, we understand that the scores are evenly spread in the distribution, resembling a normal distribution. If the median falls closer to the upper or lower quartile, there are more higher scores than lower scores in the distribution. In some box plots a different symbol may be found inside the box to represent the mean score.

With the use of lines extending from the box in both directions, called whiskers, the box plot also provides more data about the extreme scores on both the high and low end of the distribution. The whisker may or may not have a crosshatch line at the end of the whisker. The conventions for what the whiskers represent are less consistent than for the box portion of the plot. Traditionally, the whisker represents the score occurring closest to but not further than 1.5 times the interquartile value and is called a step, or the inner fence. (There also might be a whisker that illustrates the outer fence, or 3 times the interquartile range.) The whisker may be drawn to represent the lowest- and highest-occurring scores or the score value at 1 standard deviation above and below the mean. The whisker might also represent a confidence interval around the mean. Symbols occurring beyond the whiskers indicate outliers or extreme outliers, which indicate score values that are quite high or low. It is important to check the legend of the figure when interpreting a box plot to be certain what measures of dispersion are being plotted by the whiskers. In Figure 12-6 it is easy to quickly see that the variability in change of the scores for the experimental group was larger than the variability in the control group, thus the control group has a more normal distribution than the experimental group. The experimental group has more low scores than high scores, as indicated by the 50th percentile score occurring closer to the 25th percentile than the 75th percentile.

Inferential Statistics and Hypothesis Testing

Once you understand the distributions of the data on each measurement in the study, you can evaluate how researchers have chosen to test the hypotheses they used to design the study. To understand the hypotheses, you must be able to clearly identify the structure of the study: the variables and their roles in forming hypotheses to be tested.

Variables and Hypotheses

As you read the description of the methods of the study, you will be able to identify the variables, or measures that are important to the study design. There are two primary roles that a variable may play in the study design: the independent variable, or factor, and the dependent variable, or outcome. Independent variables are those that categorize the intervention that is being studied; for example, an independent variable might be generally termed "treatment," with one group receiving the experimental treatment and the other group receiving the control or standard treatment. The independent variables are set or manipulated by researchers to occur at the levels they desire, for example, two groups versus three groups. Most RCTs will have an independent variable that functions to study the intervention of interest; however, the number of levels of this independent variable, or number of subject groups, can vary, with a minimum of two groups. Time is another common independent variable found in studies of interventions in physical therapy. When the study design reports measuring the outcome variables before the intervention, after the intervention, and perhaps at one or more follow-up times, then we consider time to be an independent variable that the researcher chooses to manipulate. It is important to identify all the independent variables in the study, for these are the variables that the authors hypothesize may have caused the outcomes of the study.

The dependent variables are the tests and measures that the researchers choose to determine the effects of the intervention. If they study the effects of manual therapy versus traction on cervical spine pain for patients with osteoarthritis, then the changes in pain would depend on which type of treatment the patient received. It would be rare to find an RCT designed with only one dependent variable. It is much more likely to find many tests used, often grouped into categories, for example, impairments to bodily functions or limitations to participation and activities in daily life. Researchers may choose their outcome or dependent variables based on previous research in order to make their research comparable to other studies. Identifying the dependent variables is important to your evaluation of the statistical conclusion validity of the study because each dependent variable will be tested by a research hypothesis.

The research hypotheses are a statement of what the author believes will happen in the study to each dependent variable. For example, a research hypothesis may predict that subjects in the manual therapy group will have a greater decrease in neck pain following one treatment than will the subjects in the cervical traction group. The author will predict an outcome for each dependent variable in the study, for example, predicting that manual therapy will cause a greater decrease in pain for the experimental group as well as an increase in cervical range of motion in flexion, extension, right and left rotation, and an improvement on a functional test for the upper limbs. This goal would require six research hypotheses to be tested and the possibility for manual therapy to be favored over cervical traction on all six dependent variables or in fewer than six variables.

The study results will present the means and standard deviations measured for each group in the study on the dependent variable. Table 12.6 shows sample data. In these data, it appears that the manual therapy group showed better changes after treatment on three of the dependent variables, pain, and AROM in rotations, while the traction group had better changes in AROM in cervical flexion and extension. Some differences between groups are large and some are small. Because we can never know for certain whether these changes observed in the data between the two study groups happened by chance or were influenced by some aspect of the study design, we utilize a statistical hypothesis test to estimate how confident we are of the differences we see in Table 12.6. In other words, how likely is it that the differences between the two groups are attributable to the different treatments they received? If good research study methods, as outlined in the first part of this chapter were followed, then large differences between groups are unlikely to have happened by chance alone and are more likely to have been caused by the intervention. Therefore, each of the five variables listed in Table 12.6 will undergo a statistical hypothesis test.

There are many families of statistical tests that may be used to examine different hypotheses. Two large categories are tests of differences between means and tests of relationships between variables. Statistical hypotheses that examine differences in means between two or more groups utilize a form of hypothesis called the null hypothesis. The null hypothesis concerning differences in means would propose that there is no difference between the two group means, thus no true change has happened or what might be called a null event. For example, in Table 12.6, the change score for pain for the manual therapy group was 4 points and the change score for the traction group was 2. The null hypothesis to test this variable would state that although these two change scores are not the same, their difference is so small that it could have happened by chance. An appropriate statistic, in this instance a paired t-test, would be reported by the researcher to test this null hypothesis of no difference between groups. If the results of the hypothesis test are such that the null hypothesis is not likely to be true, then an alternative hypothesis is considered to be true. In this case, the difference in pain scores between the manual therapy group and the traction group are found to favor a better outcome for the manual therapy group. This would be reported as a statistically significant difference between group means. However, if the statistical test found that the size of the difference was too small to know if chance variations might have caused the difference, then the outcome of the statistical test is said to be nonsignificant.

Hypothesis testing allows us to calculate how certain we are about the comparisons we wish to make. The reported results of statistical tests often include one or more of the following statistics: the observed value of the specific statistic used, the probability statistic (p) that estimates of how likely it is that the results of the hypothesis test should be attributed to chance rather than the intervention being tested, or a confidence interval surrounding the difference score. A p value accompanying a statistical test that is lower than.05 is often considered a reasonable cutoff for attributing the outcomes of the study to the intervention, rather than to chance. A p value of.05 is interpreted as a likelihood of 5 in 100 of observing a difference between groups this large by chance alone—a rare occurrence. Here is another way to interpret a p value: if this study were repeated 100 times, with 100 different subject groups, the likelihood of finding the null hypothesis of no difference between groups is 5 times, while the likelihood of finding a true difference between groups is 95 times. A mean difference between groups may be presented with a confidence interval, for example, 2 (1.5–2.5). If the 99% CI around a mean difference score does not include the null value 0, this increases our confidence that the group differences are large enough to be attributable to the study interventions, for it is quite unlikely that the true difference score is zero.

Evaluating Outcomes: Effect Size and Number Needed to Treat

In evaluating the outcomes of the study, we can ask the question, were the groups similar at the end of the clinical trial? You will recall that several good research methods are used to ensure that the groups are similar at the beginning of the study, for example, random assignment of subjects to groups. The two groups illustrated in Table 12.6 appear to be quite similar on the dependent variables of interest to the researcher before the treatment was given. A comparison of the groups' pre-test scores is often performed, with a statistical test, to assure the reader that the randomization process worked and that the two groups are in fact similar before the intervention is given. From such comparisons of the equivalency of the groups prior to intervention, the researcher hopes to fail to reject the null hypothesis of no difference between groups. In doing so, one can conclude that the differences between groups on the dependent variables is not large enough to represent a statistical difference. When this is the case, any group differences found to be statistically significant at the end of the intervention can be more easily understood.

If the groups are statistically different at the end of the study on one or more dependent variables, our next question is, how different were they? This is a conclusion that readers can develop based on their clinical experience. Is a decrease in 4 points on an 11-point pain scale, before and after receiving manual therapy a big effect or a small effect? The statistical analysis will tell you whether or not the mean differences/change scores between groups are likely to have happened by chance or are due to the intervention. This is rather like a pass/fail decision; the hypothesis test either shows statistical significance or it does not. If it does show statistical significance, the reader can also benefit from interpreting a statistic that gives a sense of how large the differences, or effect of the treatment, were found to be.

The effect size is a statistic that nicely communicates exactly what its intent is: to tell you the relative effect of the tested intervention. The effect size statistic is calculated by dividing the difference between the two group means, or the difference between pre- and post-intervention means, by a pooled standard deviation of the two means.²⁰ For example, in Table 12.6 we find that the after-treatment mean pain score for the traction group was 6 and for the manual therapy group it was 4. The standard deviation for both groups was approximately 2. The ES statistic for pain scores would be calculated as 6 – 4/2 = 1. An ES of 1 tells us that the relative change caused by the intervention was approximately equal to 1 standard deviation in pain scores. Since the standard deviation tells something about the variability in the data for pain scores, evaluating the magnitude of the change score against the typical variability in the group's pain scores helps us understand how big an impact, or effect, the manual therapy treatment had. If we examine the effect of both treatments, comparing pre-tests to post-tests, we see that the effect within the traction group was 1 while the effect within the manual therapy group was twice as large, with an ES of 2.

Another statistic, the number needed to treat (NNT) can be useful in interpreting the magnitude of a comparison of treatments in a RCT.^22,23 This statistic is calculated by identifying the percentage of successful outcomes in the control group versus the experimental group, calculating the difference, and dividing that difference into 1 (this formula is described later in the chapter).The resulting number provides an estimate of the number of patients who would need to be treated, in order to have one more successful case than if you had used the control or comparison treatment. For the NNT analysis, the data must be dichotomous, for example, successful outcome versus failure. Any continuous data can be translated into this dichotomous format by selecting a cutof point to define success and failure. A NNT of 1 would indicate that every patient treated with the experimental treatment will have a successful outcome, while a NNT of 7 would indicate that you could expect to treat 7 people before you increased your success rate by 1 person, with the new treatment compared to the old treatment. A NNT of –1 would indicate that in every case, the control treatment will produce a better outcome. The NNT statistic is helpful in comparing the relative effectiveness of competing treatments, such that the treatment with the lowest NNT should be considered preferable, given similar costs and side effects of treatment. This statistic is not yet found with regularity in the physical therapy literature, but it is thought to be a clear and intuitive statistic for clinicians to use to compare clinical practice alternatives.

Power

As you read the results section of a paper, identify each hypothesis test that is reported and find in either table or text the result of the test. Your focus should be on identifying the statistically significant findings, as those are the ones on which the author will focus. If not all dependent variables measured in the study are found to have statistically significant changes in the desired direction, the author will discuss this finding in the paper, identifying perhaps methodological issues that precluded the anticipated results. One of the most common reasons identified for failure to find statistically significant results has to do with the power of the study. The power of the study is an expression of the capability of the study to find statistically significant results of a certain desired size when in truth there is a difference.¹⁹ Researchers can use simple calculations to determine the power of their study to find differences between treatments. The calculation requires knowledge of the typical variability in the primary dependent variable, an estimate of the desired effect size of interest to the researcher, and the sample size. By adjusting these elements the researcher can plan for sufficient subjects to assure the statistical conclusion validity of the study. If the researcher does not allow for a customary number of dropouts in the methods and the study is plagued by large attrition rate, the power of the study could be compromised. It is not uncommon for a study to be underpowered for examination of all important variables.

As we evaluate the power of a study, we use one or more criteria that indicate how small a change, or difference between groups, we wish to be able to find in the study. Two terms with unique definitions may be provided by the author: the minimum detectable change (MDC) or the minimal clinical important difference (MCID). The MDC is defined as the smallest change in score that can be statistically detected beyond random error.²⁴ The MCID is often identified by the researcher as a sufficiently large change score between groups that would justify a change in practice. Both of these values can be calculated from data found in previous studies of the intervention of interest (for example, the mean change score or multiplying the desired effect size by the standard deviation) using one of several formulas. For instance, if a study of manual therapy compared to cervical traction to reduce cervical spine pain found a mean change score in the experimental group of 3 points, an investigator may set 3 points as his MCID for a study of the effects of manual therapy in patients with lumbar spine pain. These statistics are useful to understand in reading studies, as they alert the clinician to the likely magnitude of changes that might be expected from the intervention, and they show that the author designed the study with sufficient power to find those differences, if, in fact, they existed.

Summary of Statistical Conclusion Validity

We have covered some of the basic concepts that will allow the reader of a randomized clinical trial to comprehend the basic statistical approach to the results section of the study. We focused on the principles of data analysis that will permit you to examine the data presented in the study, read the tables and figures, and confirm what the author provides in the text. An understanding of concepts such as variability within the data in one group or between two or more groups helps you to accept the statistical validity of the study. If you wish to master additional statistical concepts, you will find some recommended resources at the end of the chapter.

DETERMINING VALUE IN STUDIES OF DIAGNOSTIC/PROGNOSTIC ACCURACY

As clinicians we require research evidence to allow us to select the best interventions for our patients; for example, evidence that will support the choice of manual techniques, the dose of exercise, the timing of treatment, or the decision not to treat at all. But these intervention choices are only useful in the context of our clinical judgment about the patient's diagnosis and prognosis, important first steps in the patient management model. So we must also seek good research evidence to support our diagnostic and prognostic decisions. Diagnostic accuracy studies focus on evaluating clinical tests against the best available gold standard tests to determine how accurate our clinical tests are at finding patients that have and do not have a certain diagnosis.⁴ Studies of patient prognoses are designed to learn the likely course of recovery for patients in similar groups, with or without certain characteristics, for example, those patients who are likely to reinjure a joint if they return to sport with or without a protective brace. In physical therapy literature, we have experienced a significant increase in diagnostic accuracy studies, but there are fewer studies that help us understand a patient's prognosis, especially the long-term outcomes of our care. The criteria for good research practice are similar for both these types of research questions, so we will focus on diagnostic accuracy studies in this discussion. As with the criteria for determining value in the previous section on intervention studies, the goal is to avoid the influence of bias in the design and analysis of these studies. We will discuss three categories of good research practice: the selection of research subjects, the selection of measures and procedures, and the statistical estimates.

Did the Selection of Subjects Avoid the Introduction of Bias?

The methods used in observational studies to assess the diagnostic accuracy of clinical tests can have a large impact on the truthfulness of the statistical estimates for the examination. The research design options for observational studies is primarily defined by the selection of subjects: cohort studies identify subjects before any exposure has happened and follow the subjects forward; case-control studies identify subjects with the disease and those without the disease and retrospectively evaluate their exposure to the variable of interest; and cross-sectional designs select subjects and perform tests at the same point in time. Our first set of quality criteria will then focus on the research design selected or how the subjects were identified.

The selection of appropriate research subjects is as important in ensuring quality in diagnostic accuracy studies as it is in clinical trials. The investigator will carefully identify the target population of patients and establish inclusion and exclusion criteria that provide a group of research subjects who are likely to need the diagnostic test under study. Subjects who have no injury at all or are so clearly impaired as to make a diagnosis obvious should be excluded from diagnostic accuracy studies.²⁵ If these subjects are not excluded, the results of the study will be biased, usually overestimating the accuracy of the test. If the study fails to have a heterogeneous sample of subjects with a range of severity of the suspected impairment, spectrum bias may be introduced.^26,27 Spectrum bias occurs when the sample selected for the study contains subjects at the furthest extremes of the diagnostic spectrum, while ignoring subjects in the middle of the spectrum. Spectrum bias in diagnostic accuracy studies has been linked to the selection of a case-control research design. Boyer et al²⁸ performed a systematic review of diagnostic studies of clinical examinations for diagnosing carpal tunnel syndrome, finding spectrum bias in 65% of the 23 highest-quality studies, all of which used a case-control research design and were determined to have overestimated the accuracy of the tests.

To judge the adequacy of the sampling and research design, the author must provide a detailed description of the sampling methods and of the subjects' participation in the study so that you know the number of dropouts in the study. A flowchart similar to what you would expect in a RCT would be helpful.

Did the Selection of Measures and Procedures Avoid the Introduction of Bias?

The research design preferred for diagnostic accuracy studies is cross-sectional, requiring one group of subjects who receive two or more tests in a closely determined time frame. Enrollment of subjects into these studies is usually sequential and prospective, but some retrospective studies of diagnostic accuracy are reported. These studies are of less value to the clinician because so many of the following procedural aspects of the study cannot be controlled. A diagnostic accuracy study will be performed to assess a new or existing test, the index test, against a reference test or gold standard. In physical therapy research, the reference test is assumed to be the most accurate test or group of tests available to diagnose an impairment in body structures or functions or in participation in desired social activity. Any differences between the index test and the reference test are assumed to be attributable to errors in the index test.^29,30,31 It is important that the index test not be considered a part of the reference test, for example, choosing to compare a subtest score on the SF-36 to the total score on the SF-36. (The SF-36 is a standardized measure of health-related quality of life with subsets of items that provide scores in eight separate health domains as well as the composite score.) When this is done, it is likely the results will over-estimate the accuracy of the index test.

The procedures used to administer the tests in the study are also very important to the value of the study. The first aspect to consider is whether all subjects received both the index test and the reference test. If the reference test is expensive or could cause the subject discomfort or other risks considered inappropriate for that subject, the result is that a nonrandom group of study subjects receive both tests, and this could introduce a selection bias into the findings. This is less of a concern for diagnostic accuracy studies in physical therapy, but if not all subjects can receive the reference standard, the best research practice would require a random sample of study subjects to be selected for the reference test. This practice will also control for additional selection bias if only subjects who receive a certain score on the index test are given the gold standard test.

The timing of the two tests is also a criterion on which to evaluate the study. If the measures are not taken in close proximity, there is a chance of finding disease progression bias. This can occur when the subject's status changes, for better or worse, in the intervening time between measures. This must be evaluated for each study, taking into consideration how likely it is that the subjects' status might have changed. If the measures are assessing physiological properties like heart rate or blood pressure, the index and reference measures should be taken at the same time or very close in time. Other diagnostic clinical tests use imaging results as a reference standard and may be taken days apart. The author should carefully describe the decisions for the timing of the measures in these types of studies, and you will find value when you believe that the phenomenon being measured has not changed between index and reference testing.

Diagnostic accuracy studies should contain detailed descriptions of how each measure was performed, including figures or photos if helpful. This is important both for replicating the study findings or using the tests in your clinical practice and for your assessment of the appropriateness of the test. For tests of physical performance or assessment of joint structures, knowledge of the tester's position, the subject's position, the direction and magnitude of force application, and many other aspects of the test will be crucial to your assessment of the value of the test to your practice. It is also important to evaluate who performed each test, to assure that the testers were blind to the outcomes of the reference test if they were taking the index tests, and vice versa. The tester who takes the first measure will not be at risk for knowing the second test results, but even if the order of testing is randomized, the author should state the procedures taken to ensure that each tester had no knowledge of the other test results for subjects.

Were the Statistical Estimates Developed and Presented Without Bias or Error?

The next elements to evaluate in a paper reporting the diagnostic accuracy of a test are the results and estimates provided for each test. If the previously discussed methods of procuring the test results from the correct subjects have been followed, the next step is to interpret the results of the statistical analysis used by the authors. There are a variety of statistics used to analyze the data on the index test. We will define several of the most common and discuss the interpretation of each. Descriptive data (for example, frequency counts) for the subject's scores on the two tests will provide the first level of analysis of the accuracy of the index test.

The simplest way to examine diagnostic accuracy is to dichotomize the results of each test as either positive or negative. This will occasionally require collapsing data from a test that has more than two categories of outcome, for example four grades of tendon lesions might be collapsed by denoting the highest three grades of injuries as positive tests and the lowest grade as a negative test. When the test scores are not categorical but continuous data, a cutoff point must be selected to indicate when a positive test should be recorded. Table 12.7 provides a description of a typical 2 × 2 cross tabulation table that provides this first level of analysis for the results of diagnostic accuracy studies and the basic framework that allows the construction of diagnostic accuracy estimates. Table 12.8 gives the calculations for each of the statistical estimates and a brief description of how to interpret each one.

The basic diagnostic accuracy statistics allow the clinician to select tests and measures with the best statistical conclusion validity. We would desire tests that have both high sensitivity and specificity such that incorrect classification of patients is minimized, but few tests would fall into this category. The sensitivity of a test is perhaps the easiest statistic to understand, for it means that the test items were sensitive enough to identify individuals who fall into the true positive category, a positive test, and actually have the diagnosis. It is useful to have screening tools with high sensitivity, for if a patient tests negative on a highly sensitive screening tool, you are more confident in ruling out that diagnosis. A mnemonic for this situation is SNOUT, i.e., highly Sensitive test, Negative test result, rule the diagnosis OUT. The specificity of a test describes its ability to find individuals in the true negative category, a negative test, and do not have the diagnosis. Highly specific tests are good at finding individuals who do not have the diagnosis, so a positive test result on a highly specific test tells the clinician to pay attention to that result, for it may rule in the diagnosis. The mnemonic for this situation is SPIN, i.e., highly specific test, Positive test result, rule the diagnosis IN.^26,32

The positive and negative predictive values (PPV and NPV) are proportions that can be calculated from the 2 × 2 table by working the table "horizontally" rather than "vertically," as is done to calculate sensitivity and specificity.²⁰ As with the sensitivity and specificity statistical estimates, we also wish for high PPV and NPV. However, these two statistical estimates are influenced by the prevalence rate of the diagnosis for all subjects in the study. The prevalence expresses the percentage of the population of interest who have the diagnosis at any given point in time.²⁰ If the prevalence of the diagnosis is high in the study sample, this increases the predictive value of a positive test and decreases the predictive value of a negative test, and the converse is true if the prevalence of the diagnosis is very low in the study sample.

Likelihood ratios give us a proportion of subjects (with either a positive or negative test) that contrasts those who really have the diagnosis versus those who do not. For example, a likelihood ratio of a positive index test examines just those who tested positive on the index test, making a ratio of those with the diagnosis in the numerator and those without the diagnosis in the denominator (those with diagnosis/those without diagnosis). You can also think of this as a ratio of true positive subjects (TP) to false positive subjects (FP). It is clear, then, that we wish a positive likelihood ratio statistic to be high for an index test. Likelihood ratios greater than 10 or less than 0.1 are considered to be characteristic of tests that are quite strong.

Other Statistics Found in Observational Research

We will conclude this section with a discussion of additional statistics terms which may be found in observational studies that address either diagnostic accuracy or the impact of an intervention. Some simple statistics calculated from frequency data that are helpful to understand include ratios, proportions, and rates.²⁰

Ratios are created by dividing one frequency by another and are expressed using a colon (:). Proportions are calculated by dividing a subset frequency by the total frequency. Rates are calculated by expressing a proportion over a period of time. We will illustrate these statistics with hypothetical data for mechanisms of glenohumeral joint dislocation.

Your outpatient physical therapy clinic provides coverage for the sports teams at three area high schools, enrolling 800 student athletes. In the past year, you treated 40 student athletes with a diagnosis of glenohumeral joint dislocation, 38 of which were anterior in direction and 2 of which were posterior dislocations.

• The ratio of anterior to posterior dislocations is 38/2 = 19:1. There were 19 anterior dislocations for every posterior dislocation patient.

• The proportion of student athletes who experienced an anterior dislocation is calculated as 38/40 = 0.95 × 100 = 95%.

The rate of occurrence of anterior glenohumeral dislocations is calculated as 38/800 = 0.0475; and the rate of occurrence of posterior glenohumeral dislocations is calculated as 2/800 = 0.0025. These small decimal values are often multiplied by a constant to express the rate in easier-to-understand terms; for example, if we multiply the rate of anterior dislocations by a constant of 1000, the reported rate of anterior dislocations would be 47.5 student athletes in 1000 in the past year.

If the study examines how proportions may differ for groups of subjects, the calculation of risks and odds may be used. Suppose we chose to follow the 40 athletes in our practice who experienced shoulder dislocation because we wish to understand the influence of their rehabilitation on the occurrence of redislocation of the shoulder. Buteau et al have described a high prevalence of reoccurrence of glenohumeral dislocation in athletes in their case report on the use of a new piece of exercise equipment, the body blade, which oscillates to provide resistance in upper-limb strengthening programs.³³ Looking retrospectively at the experiences of these injured athletes, we find the data reported in Table 12.9. Surprisingly, one-half of the injured athletes were not referred for rehabilitation, but were only immobilized in a sling; and the other half were referred for physical therapy, including the use of the body blade! The frequency data in this 2 × 2 table can be translated into several statistical estimates: risk ratios, absolute risk reduction, number needed to treat, and odds ratios. It is not common to find a 2 × 2 table presented in a research report; however, these statistical estimates are often used to describe how the groups differed.

One can quickly see that the risk of reinjury to the athlete's shoulder was higher for the immobilization-only group than the experimental/body blade group. A calculation of the absolute risk reduction allows the subsequent calculation of the number needed to treat; in this instance, it is quite low at 2.22, indicating that you would need to treat only two athletes with shoulder dislocation with the body blade to decrease their risk of reoccurrence of their dislocated shoulder. The odds ratio is constructed from the odds of a person in the immobilization group having or not having a second dislocation; in this case the odds of the control subjects having a second dislocation are much higher than the odds that they did not have a second dislocation. The odds ratio of 8.5 indicates that a person with a second dislocation is 8.5 times more likely to have been in the immobilization group than in the body blade group. Odds ratios are often presented with a confidence interval, so you can observe the upper and lower bound for this estimate. Odds ratios are also used to indicate the magnitude of relationship between more than two groups and an outcome variable, and in this type of analysis, one of the groups is used as the reference standard, and odds ratios are calculated for each of the other groups in reference to that group. For example, if we seek to understand the impact of age on balance, and we group our subjects into four age groups, odds ratios may be presented to express the likelihood that the groups have different balance performance.

CONCLUSION

This chapter has provided an overview of important elements that should be considered when assessing the value of experimental and observational research as it is used to help determine the accuracy of diagnostic and prognostic measures and the worth of interventions for patients. The primary focus in discussing each of these elements has been to help assure that the researchers have done what can be done to reduce the risk of bias in conducting their research and in presenting the results. It is only through such efforts that you can determine the level of confidence you should have in applying the results of the study to patient care. These elements were discussed in the context of an individual study, but they apply to all of the sources of evidence that we will discuss in subsequent chapters.

SELF-ASSESSMENT

1. Find an article that supports an intervention provided to Mr. Ketterman. Identify each of these features in the article:

   1. Design

   2. Methods intended to reduce bias in selection and management of subjects

   3. Methods designed to reduce bias related to selection of interventions and measurement of outcomes

2. Using the same article, identify the information that would determine external validity.

3. Using the same article, identify the descriptive and inferential statistics. Do they support the conclusions reached by the authors?

4. Find an article that supports a diagnostic/prognostic test used in Mr. Ketterman's care. Identify each of these features in the article:

   1. Methods intended to reduce bias in selection and management of subjects

   2. Methods intended to reduce bias in the selection of measures and procedures

5. Select a diagnostic/prognostic article that presents ratios, proportions, and rates. How can you use these data as you make decisions about Mr. Ketterman?

CONTINUED LEARNING

• There are some excellent texts that provide more detail about research design and methods in physical therapy. They include:

DiFabio R. Essentials of Rehabilitation Research: A Statistical Guide to Clinical Practice, Philadelphia: FA Davis; 2013.

Domholdt E. Rehabilitation Research: Principles and Applications, 3rd ed. St. Louis: Elsevier Saunders; 2005.

Fetters L, Tilson J. Evidence-Based Physical Therapy, Philadelphia: FA Davis; 2012.

Jewell DV. Guide to Evidence-based Physical Therapist Practice, 2nd ed. Ontario, Canada: Jones and Bartlett Learning LC; 2011.

Portney L, Watkins M, Foundations of Clinical Research, 3rd ed. Pearson Prentice Hall, 2009.

1. Select and become familiar with one of these texts so you can use it to explore questions you have as you read the evidence.

• Internet search sites can also be useful for learning more about specific methodological decisions and statistical tests. Select terms for articles as you are reading the evidence and explore what you can learn about them through Internet searching.

REFERENCES

INTRODUCTION

Evidence based practice is built on a process that begins with a clinician's need to know something in order provide the best care to the patient. The clinician translates the information needed into an answerable question and then efficiently finds the best type of evidence related to the question. In this chapter we will focus on determining the utility of individual studies, the base of the 6S pyramid described by DiCenso et al.^1,2 shown in Figure 13-1, in helping to make good clinical decisions. Since individual studies are the most prevalent form of evidence that clinicians use, it is important to know how to assess them. We will provide the reader with the most currently used criteria, based on the principles discussed in Chapter 12, to assess the value of evidence derived from experimental and observational research.

STUDIES

As Schardt tells us in Chapter 11, as of 2011, there are over 17,000,000 individual articles referenced in Medline, so it is clear that the individual study is the most common form of evidence available to clinicians. In addition, most of these studies are published in journals that use masked peer review. As we discussed in Chapter 10, this means that experts in the field have reviewed the authors' report of their research methods and findings and have found them acceptable for publication, without knowing who the authors were. These studies can be found at all of the many sources reviewed in Chapter 11.

Studies Focused on Physical Therapy

Physical therapists can and should find articles of interest in all major health care literature. Journals with a broad focus, such as the New England Journal of Medicine (NEJM), the Journal of the American Medical Association (JAMA), the British Medical Journal (BMJ), journals with a specialty focus such as the Journal of Bone and Joint Surgery (JBJS), Neurology, Pediatrics, Journal of the American Geriatric Society, the Archives of Internal Medicine, and journals that focus on rehabilitation, such as the Archives of Physical Medicine and Rehabilitation, can all provide many articles that address the care physical therapists provide to their patients. In addition, there are now many journals published all over the world that focus specifically on physical therapy. Box 13.1 shows a list of many of these journals.

The articles in these journals cover a wide range of topics, but the ones of specific interest to physical therapists primarily fall into two groups: studies that address diagnostic or prognostic accuracy and studies that assess the efficacy and effectiveness of interventions. The standards set by peer-reviewed journals vary, and, of course, the generalizability of even the best research will vary based on the patients in question; therefore it is imperative that clinicians review studies carefully to determine their value to their clinical decision making. There are several published checklists that can be used to evaluate single studies. In addition to the CONSORT checklist^3,4 and the PEDro scale⁵ used for RCTs (randomized controlled trials), the STARD checklist^6,7 and the QUADAS instrument^8,9 can be used for studies of diagnostic accuracy, and the STROBE scale can be used to evaluate observational studies in epidemiology.¹⁰ Olivo has performed a systematic review of a range of criteria used to assess RCTs in medicine, finding no single scale that is recommended for physical therapy research.¹¹ Using the principles laid out in Chapter 12, we have developed a set of questions that you can use in reading and assessing articles.

ASSESSING STUDIES OF DIAGNOSTIC/PROGNOSTIC ACCURACY

The first steps in patient care management are to determine an accurate and useful diagnosis and to determine the prognosis of the patient with that diagnosis. If we have not done this accurately, then we cannot begin to determine appropriate interventions, as the success of interventions is linked to their application for certain diagnoses. It is useful to be reminded that physical therapists are most often making diagnoses of impairments and dysfunction. As we discuss aspects of diagnostic accuracy studies, we will use a checklist of questions designed to help you to determine whether or not good research practices have been used to eliminate bias or errors and control for variability in these investigations.^8,9 Box 13.2 summarizes the recommended questions to use to determine the value of a study of diagnostic accuracy. These questions can be used in a formal way by completing a checklist, or more informally by having them in mind as you read the article.

As discussed in Chapter 12, studies of diagnostic or prognostic accuracy generally use observational designs. This is because these studies ask questions fundamentally different from those addressed in interventional studies. For example, in diagnostic accuracy questions, cross-sectional studies are common so that it is possible to distinguish whether the test can accurately select those subjects who really have the disease. In prognostic studies, longitudinal studies are common so that the prediction ability of the test can be determined by what actually happened to the subjects over time. Because these designs do not offer some of the protections against bias that can occur in a controlled experiment, it is imperative to look carefully at the other mechanisms the researchers used to reduce the potential for bias.

Therefore, it is important to focus on the sample of subjects used in these studies. So the first questions address the appropriate documentation of the sample.

• Did the sample selected for the study represent an appropriate range of subjects who might require the test?

• Did all the subjects take all the tests?

When the researchers do not meet these criteria, they should offer information about why they were not able to do so. This allows the reader to determine how much these issues limit the usefulness of the study.

It is also important to have clear information about the tests being assessed in the studies. In some situations there is a gold standard, a test that is known to have very good discriminative ability. Perhaps the study is designed to determine how a cheaper or more clinically practical test, referred to as an index test, compares to this gold standard. In some other cases, there is no gold standard but there is a clinical standard, a test that is widely used, and the study is designed to see if the new index test might actually be an improvement. The management of the measures used in a diagnostic accuracy study can be assessed with these questions:

• Is the gold standard test likely to correctly identify a subject?

• Is the index test a part of the reference test?

The researchers also need to use careful procedures designed to reduce the effects of the variability that can occur in conducting and interpreting the tests. The researchers should have put into place the same type of procedures used in experimental designs to blind data collectors to the status of the subjects they are evaluating.

• Did all the subjects receive both the index test and the reference test? If not, did a random sample of the subjects receive both?

• Was the timing of the index and reference measures sufficiently close?

• Were the tests described with sufficient detail?

• Were the testers blind to the results of the preceding test?

Finally, the way the researchers managed and reported the data needs to be assessed. Many clinical diagnoses are based on interpretation of test results. What clinicians already know when they look at the next data point can influence how they interpret that data point.¹² Therefore, it is important that the clinical data is managed in the study just as it would be in clinical practice. In addition, all test results, even if they are not totally clear, need to be included if accurate sensitivity and specificity are to be calculated.

• Were the same clinical data available in the study as would be available when the test is used in practice?

• Are all test results reported, even if they are not clearly interpretable?

If all of these questions can be answered with a yes, then the clinician can have confidence that the researchers have designed their studies in a way to minimize the effects of bias or variability on their results (validity) and can move on to determining if the results can be appropriately applied to individual patient care (generalizability).

ASSESSING STUDIES OF CLINICAL INTERVENTIONS

When you are looking for evidence that will help you select the best intervention for your patient, you will likely search for studies that test the effectiveness of various treatments on certain outcomes that are of importance to your patient. The randomized controlled trial (RCT) is considered a strong research design for such a question. We will use this research design to illustrate the criteria by which you will determine if a study of interventions is of value to you. The criteria are broadly useful to critique studies that use a variety of research designed to understand the efficacy and effectiveness of interventions. Box 13.3 contains a checklist to be used to evaluate the research validity of a clinical trial.

The first item in the checklist reflects our decision about the rigor of the research design selected by the investigator for the study:

• Was the research design appropriate for the research question? Did the researchers use the strongest possible design to demonstrate a causal relationship between the intervention and the outcome?

Chapter 12 provides information about many research designs. As we discussed there, the best design to demonstrate a causal relationship is the RCT, with its three essential features of random selection or allocation of research subjects to groups, the inclusion of a control group for comparison to a treatment group, and purposeful manipulation (dose, frequency, or duration) of the treatment variable. Sometimes the researchers cannot achieve each of these fully, so it is important for the clinician to recognize the limits of the study and the other actions taken by the researchers to provide useful, trustworthy data.

Descriptions of the sample and its selection are important because they contribute both to the trustworthiness or validity of a study and to the clinician's decision about the generalizability of the study to the clinician's patients.

• Were the subjects chosen for the study the right ones?

• Were the subjects randomly allocated/assigned to treatment and comparison groups?

• Were the groups similar at the beginning of the study on measures that matter to you?

• Did few subjects drop out of the study?

• Were the subjects blinded to which group they were assigned?

As you read the study, ask yourself how well controlled the study was in regard to the selection and treatment of subjects. Consider if the description of the subjects provides you with the information you need to decide if you can apply the findings to your patients. Because of the nature of physical therapy, subjects often cannot be blinded to the group to which they are assigned, so it is important to be sure that at least the evaluators are blinded.

The information about the specific methods used in the study is also important for trustworthiness and generalizability.

• Were the interventions selected for each group clearly described and clearly different?

• Did subjects receive a sufficient dose of treatment?

• Was the clinician giving the interventions blind to the assignment of group members?

As you read the study intervention procedures, you will use your clinical reasoning and expertise to evaluate each of these methodological decisions. Do they make sense to you, and can you imagine that these are the right decisions to provide the best opportunity for finding an effect for one type of exercise over the other? Or does your experience tell you that one or more of the researcher's decisions are likely to have introduced bias or errors? Recall the example used in Chapter 12 about a research study designed to investigate whether eccentric or concentric quadriceps strengthening exercise will produce better outcomes for patients who received a total knee arthroplasty. Your clinical judgment about the methods can tell you, for example, if sufficient time was allowed for the intervention to have an effect. Were the subjects in the concentric group clearly required to work less than those in the eccentric exercise group? Because of the nature of physical therapy, the physical therapist giving the exercise interventions is very likely to know to which group the subject was assigned. If so, do the methods identify ways that the evaluation function is separated from the practitioner function and that the evaluator is blind to the assignment? In addition to your clinical wisdom, you would hope to find the authors describing the evidence in the literature that informed their decisions about each step of the procedures.

Next we need to consider carefully the decisions made by the researchers about the tests and measures selected to demonstrate the differences in the results of the intervention across the groups in the study.

• Were the right tests and measures used? Ones matched to the purposes of the study, across a range of outcomes; ones that are reliable and valid?

• Were the observers who took the measurements blind to the group assignment of the subjects?

• Was the timing of the measures appropriate?

Often the tools are not ones that are suitable for use in the clinic, but we still need to use clinical judgment to determine if the measures address outcomes that are clinically meaningful for this patient population. Are the tools able to reliably and validly identify a clinically meaningful difference? Does the timing of the measures match what we know clinically about healing and recovery times for our patients? As we have mentioned above, it is very difficult to have subjects and providers of the intervention blinded to assignment, but it is imperative that the observers or evaluators are.

The final criteria we suggest for our checklist of determining the value in a RCT are related to the decisions made about the election and interpretation of statistics and the reporting of the outcomes.

• Were all subjects who started the study accounted for at the end of the study?

• Were all subject data analyzed in the groups to which they were assigned?

• Were the groups similar at the end of the trial? How much did things change for each group and between groups?

First, we examine the issue of subject attrition in the study. Previously, we included criteria to assess whether there were few subjects who dropped out of the study. We return to this concern because it can greatly affect the statistical analysis if this happened in the study. The author should account for the participation of all subjects who started the study, and those who, for example, completed one post test versus all follow-up testing sessions. The PEDro scale for assessing value in a RCT requires that data on an important outcome of the study was reported from 85% or greater of the subjects assigned to the group.⁵ It is particularly important to assess whether the subject attrition happened differently within the study groups, for if one group lost 2% of subjects and the other lost 20% of subjects, the groups may no longer be as equivalent as they were at the start of the study. In clinical trials with several measurement points, it is usual to lose some subjects, so a method of analysis has been developed to address this problem, called intention to treat analysis. This analysis requires the researcher to analyze the subjects' data in the groups to which they were originally assigned, rather than analyze the data including changes in study protocol or deleting data for a subject who dropped out of the study. These occurrences can provide a biased statistical analysis by only including subjects who completed every aspect of the study, while dropping from the analysis subjects who were not responding to the intervention and chose to drop out. There are several methods of handling these problems in the data statistically, and the authors should describe for the reader exactly how they handled missing data and attempted to analyze all subject data in the originally allocated groups.

The last criterion stems from the statistical analysis of the magnitude of the effect of the intervention, when compared to a control or contrast group. As you read the study results, you will form opinions as to the importance of the differences the authors found, both within each group over time and between the groups. If this criterion is not met to your satisfaction, either because of a poor analysis or because the study was not sufficiently large to detect a minimal clinically important difference (MCID), it is not likely that the study will have any value for your patients. Thus it is appropriate that the final criterion is the most important: how big a difference did this intervention make? Is this difference large enough for you to decide to use this intervention, either for a particular patient or as a change in your typical practice for this patient population?

CONCLUSION

In this chapter we recommend criteria that will assist you to critique the various types of research that can inform your decisions with your patients. As you evaluate a study for any given clinical decision, you may find that you weight one or more criteria more heavily, and this practical approach to evaluating the value in the evidence makes good sense. We have recommended a total of 15 questions that can guide you to your decision about the value to you and your patients of any single RCT. Does a study need to meet every criterion? Few will meet every criterion as tightly as you might wish, for clinical research is a complicated process prone to many unexpected or unplanned events. As you evaluate a research report of a RCT, you may find some of the 15 criteria less important, given the intervention under study. Each decision you make about the value of a study may be adjusted as you read a new study with better design and controls in place. But it is very likely that you will learn something about your practice decisions from any study you critique, as the process of critical appraisal strengthens one's clinical reasoning capabilities.

In the next chapters you will learn about a variety of what we term filtered evidence—studies that have been critiqued for you. These are efficient sources of evidence and often provide stronger evidence than any one study can. However, when any single study is important for you to incorporate into your clinical reasoning, the principles discussed in this chapter will help you draw your own conclusion about the value of the study to you and your patient.

SELF-ASSESSMENT

1. Return to the articles that you used in Chapter 12, Self-Assessment, Questions 1–5, and use the questions we have presented in this chapter to assess their value to you.

2. What advice would you give the therapist who is caring for Mr. Ketterman based on your assessment?

CONTINUED LEARNING

• One of the best ways to develop better skill in an area is to teach about it. Develop a presentation to share with your colleagues that is based on the questions here. Then lead them in a regular analysis of literature that reports on care for groups of patients that you treat frequently.

REFERENCES

INTRODUCTION

There are several sources of information that we can use to help us find and organize evidence for use in daily practice. In Chapter 12 we discussed how to differentiate among individual studies and in Chapter 13 how to assess individual studies. But looking at one study at a time can be time-consuming. The amount of information that is now available is staggering. One analysis of the quantity of material available electronically shows that in 2010 there were over 988 exabytes (1 exabyte equals 1 quintillion bytes) of information available on the Internet. This is equivalent to over 18 million times the number of books ever printed.¹ There are now many examples of abstractions of individual studies that help bring some meaning out of this amazing mass of information for clinical practice. In this chapter we will discuss synopses of individual studies (Fig. 14-1).² As DiCenso et al tell us, "The advantages of a synopsis of a single study over a single study are 3-fold: first, the assurance that the study is of sufficiently high quality and clinical relevance to merit abstraction; second, the brevity of the summary; and third, the added value of the commentary."³

SYNOPSES OF STUDIES

Synopses are designed to provide clinicians with direct, quick advice about specific clinical action by providing a brief, focused abstract of a study. A synopsis can be as brief as a single sentence, such as the title of many of the review articles in some evidence based journals. A review of the titles in Box 14.1 shows that while single sentences might provide some general guidance, they usually do not give clinicians sufficient information about the specificity of an intervention to be useful in treatment planning. However, the single sentence can tell clinicians if reading the review, and perhaps the article, would be worthwhile.

The next option in a synopsis is a brief extraction of the study to support a specific conclusion. These brief extractions are usually kept to one page or less to facilitate the clinician's being able to see quickly the recommendation for action and to understand the supporting data for this recommendation. The full text of the review articles listed by title only in Box 14.1b. can be found at http://ebm.bmj.com. Although the full text does not provide the detail needed to replicate the intervention, it does demonstrate the value of the intervention as compared with many other interventions, and it does so in less than one page.

As this example demonstrates, the synopsis includes the essential information from the study that supports a specific conclusion. Synopses usually start or end with a clear statement or recommendation about a specific clinical action with a specific patient population. When appropriate synopses are available, a clinician can determine a course of action in a matter of a few minutes. As with all evidence the utility of a synopsis depends on how well it matches the clinician's question. The synopsis also still has the problem that it typically focuses on one aspect of patient management, rather than the range of decisions needed in actual care.

Published Synopses That Apply to Physical Therapist Practice

There are several sources of synopses available for physical therapists. One primary source can be the synopses created for other health care practitioners that address issues related to the physical therapist's management of patients. As we have mentioned above, there are now many journals that are designed specifically to provide synopses of the original literature published elsewhere. Box 14.2 lists a compilation of journals across the health care continuum with a primary focus on assessment of evidence in a synopsis format to help clinicians find information easily and review it quickly. In addition to journals that have this as a primary focus, many clinically oriented journals that report original research also have sections that are devoted to reviews of their own or other articles written specifically to guide clinicians. For example, "ACP Journal Club" is a monthly feature of Annals of Internal Medicine; "Current Best Evidence" is a regular feature in The Journal of Pediatrics, and "Practice Oriented Evidence that Matters (POEMS)" can be found in the Journal of the American Academy of Physician Assistants.

There has also been a growth in the number of entirely Web-based sources that focus on providing synopses of research selected from many journals and organized by specialty or patient problem. These include Essential Evidence Plus, which has grown from the original POEMS project by the Journal of Family Practice into a site where readers can find synopses of original research and of systematic reviews, as well as assistance with determining probability levels and coding patient care for accurate documentation. Another example of a Web-only resource is Journal Watch, from the publishers of The New England Journal of Medicine, that includes reviews of articles from over 300 journals and that organizes these reviews in 13 specialties and for 20 patient topics. Both sites will also provide the subscriber with a daily digest of new material as it is added to the site. (Links to active sites for these resources, and the ones described below, can be found at the F.A. Davis Web site associated with this text.)

All of these sites require a fee for the license or subscription, but there are also sites that provide synopses without charge. A very good example of these is Evidence Updates, which is a collaboration between the British Medical Journal and McMaster University. Over 120 journals are scanned for relevant articles, prerated for quality by research staff, and then rated for clinical interest and relevance by practitioners. Each subscriber then receives regular e-mails with a list of the articles with links to the article abstract.

There are several examples of synopses that exist specifically for rehabilitation or physical therapy. Each uses a different format and may have a different primary focus, but they all provide clinicians with ready access to short descriptions of studies with some assessment of their value or statement of their clinical implication.

Rehab+, part of McMaster University's evidence based practice initiative and a subset of Evidence Updates, provides selected abstracts from 130 clinical journals. The articles are pre-rated for quality by a research staff and then reviewed by a panel of practicing clinicians for relevance to practice and for newsworthiness. Each registrant to Rehab+ receives e-mail alerts with links to the article abstract (as published with the original article) along with the article's relevance and newsworthiness ratings. Figure 14-3 shows an example of the notification that each registrant receives. As of this writing, registration is free.

"The Bottom Line" is a feature of Physical Therapy that provides a translation of study findings into clinical practice for selected clinical research articles published in that journal. Each "Bottom Line" asks and answers questions about the research, such as:

• What problems did the researchers set out to study, and why?

• Who participated in the study?

• What new information does this study offer?

• What new information does this study offer for patients?

• How did the researchers go about the study?

• If you are a patient, what might these findings mean to you?

• What are the limitations of the study, and what further research is needed?

This feature is published as a companion to the article of interest in both paper and electronic format.

Collections of Synopses Relevant to Physical Therapy

There are two more advanced and organized resources available that are frequently used to conduct searches for synopses of relevant material in physical therapy. These are Hooked on Evidence and PEDro (Physiotherapy Evidence Database).

Hooked on Evidence

Hooked on Evidence is a service of the American Physical Therapy Association (APTA). It is a database of article extractions built by contributions from members of the APTA. At this writing, it includes only articles related to interventions, not to diagnosis or prognosis. Articles are selected by the reviewers or suggested by the managers of the database and specific data are extracted in an organized format. Extraction is done by choosing from among preset options in drop-down menus. The data presented for each article include:

• Design type

• Study population characteristics including number, dropout rate, and inclusion and exclusion criteria

• Blinding status

• Narrative description of the interventions received by all treatment and control groups

• List of all outcome measures

• Absolute and standardized mean differences with confidence intervals for each continuous outcome measure

• Odds ratio, risk ratio, and number needed to treat for each dichotomous outcome measure

There is also a link to the article abstract at PubMed. The articles can be found by searching keywords, which produces a list of all relevant articles, or by choosing clinical scenarios that represent typical patient populations in physical therapy. If the user chooses a scenario, the related articles are organized by design type (see Fig. 14-4a). Beginning in 2013, Hooked is available in read-only format to all users. APTA membership or an annual subscription fee.

PEDro

PEDro is an acronym for Physiotherapy Evidence Database, which is produced by the Centre for Evidence-Based Physiotherapy at the University of Sydney. PEDro also focuses on interventional questions and provides information about randomized controlled trials (RCTs), systematic reviews, and evidence based clinical practice guidelines. The contents can be searched based on therapy type, patient problem or impairment, body part, specialty practice, and by type of article (RCT, systematic review, guideline). See Figure 14-4a and b for an example of search results. All RCTs are rated by trained volunteers to identify those trials that are more likely to be valid and to contain sufficient information to guide clinical practice. The PEDro scale for this rating includes 10 items: random allocation, concealed allocation, similarity at baseline, subject blinding, therapist blinding, assessor blinding, greater than 85% follow-up for at least one key outcome, intention-to-treat analysis, between-group statistical comparison for at least one key outcome, and point and variability measures for at least one key outcome. Items are scored as either present (1) or absent (0) and summed for a total score for the study.^4,5,6 Each citation includes the score, with a breakdown of each item, an abstract, and a link to full text, when available. At the time of this writing, PEDro is free to all users through support from several physical therapy member organizations, insurers, and licensing authorities.

CREATING SYNOPSES

The intent of a synopsis, as we have said, is to make information readily available to clinicians about the clinical implications of any particular study. But, even with the many published sources of synopses there remains a vast amount of research that is of interest to clinicians that has not been synopsized. Several techniques have been developed that can be used by clinicians to produce their own synopses to fill this gap.

Appraisal Worksheets

Many evidence based practice authors and groups have developed worksheets for individual practitioners to use in developing their own synopses of individual studies. These worksheets are usually organized around the purpose of the study. For example, Jewell identifies different appraisal issues for studies about diagnostic tests, prognoses, and interventions as studied by both experimental and observational designs (see Chapter 12).⁷ These categories are very similar to the worksheets developed by the Center for Evidence Based Medicine.⁸ The Critical Appraisal Skills Programme of the United Kingdom's National Health Service has a variety of worksheets specifically designed to review the following types of studies:

• Randomized controlled trials (RCTs)

• Qualitative research

• Economic evaluation studies

• Cohort studies

• Case-control studies

• Diagnostic test studies⁹

Appraisal forms generally have similar structures. They are designed to demonstrate the worth of the study in helping clinicians make decisions, highlighting many of the issues about value that are covered in Chapter 12. The questions from the CASP qualitative research appraisal are shown in Box 14.3.

Some appraisal forms are designed to resemble scales, with assigned points for each question, which can then be summed to give an overall score. However, this single number does not give other readers much information about the reasons that an article may not have received full points, thereby causing clinicians to lose what may be valuable information. Therefore, it may be best to simply review the full scoring sheet when deciding about the use of evidence from a reviewed article.

Efforts have been made to bring more standardization to these worksheets, so that clinicians can have more confidence in their use to help guide care. This is especially needed, since so many different scales are available. One analysis showed that there are at least 21 distinct scales used in the literature to appraise work of relevance to physical therapy¹⁰ This review also showed that up to that time, while many of the scales appear to measure the appropriate issues (i.e., had good face validity), none of these scales had been tested for reliability and validity as measurement tools. The best advice at this time is for a clinician to choose an appraisal form that highlights the major concerns for the particular type of study and use the form as a guide to thinking and as a record of the analysis.

In a related effort, evidence based researchers have developed statements to guide authors of study reports about the elements that should be included in reports of particular kinds of studies. These include CONSORT for randomized controlled trials,¹¹ STARD for diagnostic studies,¹² STROBE for observational studies,¹³ and QUALRES for qualitative research studies.¹⁴ In turn, these statements are also used by researchers in designing their studies and by journal editors in reviewing reports for publication. But the authors of the statements caution that they are not designed, nor validated, to be used by readers in assessing quality of evidence.¹²

Critically Appraised Topics

When there is no worksheet suitable for appraisal, clinicians can create a more narrative synopsis. Perhaps the most familiar form of narrative synopsis is the Critically Appraised Topic (CAT), as described in Evidence-Based Medicine by Straus et al.^15,16 A CAT is a one- or two-page summary of all of the steps involved in an evidenced-based approach to the literature. CATs are generally created by clinicians for their own use, either personally or in groups. They are written because clinicians want to document their review of a specific article for future reference themselves or to share with their colleagues. Typically, a CAT is written about an interventional trial with an emphasis on RCTs; however, they can be written about groups of articles and about diagnostic or prognostic questions. The clinical question is defined, the search strategy and results are described, selected articles are cited, and details of the study are summarized, and an appraisal is done. Finally, a conclusion is given about recommended clinical behavior that reflects the strength of the methods and evidence at hand. There are several different formats for writing CATs (references). Box 14.4a shows the typical elements of a CAT. Box 14-4b shows a completed CAT written using this format.

CATs are written in response to a question that arises directly from clinical practice. As has been said, very seldom can we find a single article that exactly matches a full clinical question. Much more often the article is answering a specific part of the question. By collecting a series of CATs on the overall general topic, we can begin to bring together pieces of evidence that will eventually give us a clearer direction. Another important consideration is the nature of the clinical bottom line that serves as a conclusion derived from the study and as advice to the clinician. The bottom line is not a summary of the research findings. Instead, it is a recommendation to the clinician about action or clinical behavior. Clinicians need to take action. They can choose to adopt the new technique or assessment or they can choose to stay with the current option. They do not have the luxury of waiting for better research before treating the patient, as much as they might like to. The bottom line therefore needs to reflect the best judgment that can be made based on the evidence available in the study.

While it may seem that the growth of synopses means that clinicians do not need to develop their own, CATs continue to have many uses. They can help individual clinicians organize their thoughts about a specific article and build those thoughts for a body of literature by accumulating knowledge in a focused way. By writing down the findings, clinicians also greatly improve the ability to retrieve the work as they encounter similar patients in the future. They can also have a great impact if a group of clinicians collects CATs on topics related to the patient populations they see frequently. If a clinic with 10 staff members agreed that everyone would do one CAT a month, by the end of a year there would be 120 articles reviewed, with the information available to everyone in a very accessible manner. Finally, CATs are a great way to hone EBM skills, thereby also increasing our expertise in reviewing the materials prepared for us by others.¹⁵

CONCLUSION

Synopses are an excellent way for clinicians to find or create for themselves succinct assessment of research that focuses on the clinical implications of the work. They are increasingly available electronically, with options for "push" technology, to bring the information directly to the clinician. The majority of synopses remain focused on single studies that assess one aspect of care, generally a specific intervention.

SELF-ASSESSMENT

1. Based on the questions you developed for Mr. Ketterman, review recent issues of the journals listed in Box 14.2 to see if you can find examples of synopses that can help you in your practice decisions.

2. With a partner, identify two articles that can help you make decisions about Mr. Ketterman's care. Each of you should use the questions in Box 14.3 to create a synopsis of one article. Then share the synopsis. What are the strengths and weaknesses of using this approach to share information?

3. One way to see if you are on target in writing your own synopses would be to write a CAT for an article in Physical Therapy that also has a "Clinical Bottom Line" available online. Write the CAT before reading the "Bottom Line" and then compare your assessment with that of the expert.

CONTINUED LEARNING

• There is no better way to learn about synopses than to prepare some yourself. Ask a colleague to join you in assessing articles of joint interest and then share the synopses. This can grow to include several therapists in a practice to build a library of relevant synopses.

• Read synopses written outside of physical therapy (many of which are available in full text through public search engines) to identify the range of articles that are available to provide guidance about physical therapist practice.

• Contribute to Hooked on Evidence by performing an article abstraction.

REFERENCES

INTRODUCTION

Chapters 12, 13, and 14 have been concerned with the collection of data and its analysis from individual studies and their syntheses. These are forms of primary analysis. When trying to answer a clinical question with a limited amount of time and resources, it is common to focus on finding a few, or even only one, primary resource and therefore end up with an incomplete answer.

Most clinicians would feel more confident in applying research evidence if they could find complete and valid answers, and find them more efficiently. This is where what Haynes calls a synthesis of the relevant primary data is useful (Fig. 15-1).^1,2 A synthesis, or secondary analysis, takes the form of a systematic review, where all the data on a topic is aggregated, with or without further statistical analysis, to gain a broader picture of the issue. This solves the clinician's problem of limited time and resources. It also addresses an issue frequently found in much of health care, where it is difficult to find large cohorts of subjects, and the resultant studies have a small number of participants (ns). Such studies individually do not provide sufficient information to guide practice. A synthesis has the advantage of collectively analyzing the results from these studies with a small number of subjects and identifying possible significant results that cannot be seen looking at one study at a time.

This chapter will consider the systematic review, with and without a meta-analysis, as a way to quickly find answers to clinical questions. It will show you how to be a discerning reader of reviews and introduce you to methods commonly used to present the results. In addition, we will also discuss synopses of syntheses, Haynes's next level up the pyramid.

SYSTEMATIC REVIEWS AND META-ANALYSES

There is a long history of reviews in every discipline; these reviews are often based on the author's resources or preferences in selection of the articles to be reviewed. Often the review is designed from its inception to support a particular point of view. However, such reviews are not the best way to apply evidence from the literature to clinical decisions. In the context of evidence based practice, we want reviews that are comprehensive and free from author bias. This has resulted in the development of a particular form of review, termed systematic review. "Systematic reviews are concise summaries of the best available evidence that address sharply defined clinical questions."³ Like a good clinical study, a systematic review is prepared following a careful plan. Prestated methods for including and excluding relevant studies are used and primary studies that meet these criteria are used as "subjects" in the review. These criteria help to minimize bias and random errors.

The way in which the results are presented is called the evidence synthesis. The form that the evidence synthesis takes is based on the type of data reviewed. If the data are qualitative, then a meta-synthesis is carried out. If the data are quantitative and homogenous, then a meta-analysis is employed. If the quantitative data are not homogenous, then a narrative summary is used.⁴ As Egger et al summarize, "Well conducted meta-analyses allow a more objective appraisal of the evidence than traditional narrative reviews, provide a more precise estimate of a treatment effect, and may explain heterogeneity between the results of individual studies."⁵

Finding Systematic Reviews and Meta-Analyses

Systematic reviews can be found in journals. They can also be specified as a methodology when searching in databases such as PEDro, CINAHL, and PubMed (see Chapter 11). However, systematic reviews are most easily found as a Cochrane Review in the Cochrane Library, which can be accessed directly, through PubMed, or through gateways, such as APTA's Open Door. A Cochrane Review is a systematic review of primary research. It is considered the gold standard of systematic reviews as it must be carried out according to a standardized and rigorous methodology. Jadad et al⁶ found that Cochrane reviews are of comparable or better quality and are updated more often than reviews published in print journals. Therefore, they are a good starting point when looking for the answer to your clinical question.

There are several places to investigate within the Cochrane Library. As well as the Cochrane Database of Systematic Reviews (CDSR), the Cochrane Library includes the Cochrane Central Register of Controlled Trials (CENTRAL), with references to completed and ongoing randomized controlled trials; the Cochrane Methodology Register, with references to methodologic papers; and three other databases of systematic reviews (Database of Abstracts of Reviews of effects—DARE), health technology assessment reports (HTA), and economic evaluations (HEED).⁷

Assessing the Quality of Reviews

The definition of systematic reviews, including meta-analyses, as provided by Chalmers and Altman⁸ is, "a review that has been prepared using a systematic approach to minimizing biases and random errors which is documented in a materials and methods section." [Emphasis added] This seems a straightforward definition when, in fact, there are myriad sources of error and bias. Since these syntheses are so useful to clinicians seeking answers for their clinical questions, several appraisal tools have been developed to help clinicians recognize errors and biases, assess the value of the synthesis, and summarize the results from the synthesis, so that they can make good decisions about applying the results of other views. As we saw in discussing synopses of studies (see Chapter 14), evidence based researchers have developed statements to guide authors of systematic reviews and meta-analyses about the elements that should be included in these types of syntheses. The most well known of these statements is PRISMA (http://prisma-statement.org/statement.htm). In turn, these statements are also used by researchers in designing their studies and by journal editors in reviewing reports for publication. But, we will use the appraisal tool developed by the Critical Appraisal Skills Programme (CASP), which is a program within the Learning & Development Unit of the Public Health Resource Unit of the United Kingdom's National Health Service, as it is specifically designed for the reader of systematic reviews.

Ten Questions to Help You Make Sense of Reviews

The tool, 10 questions to help you make sense of reviews, is a popular and easy-to-use appraisal tool that provides a reader with an efficient and effective screen for determining the quality of a systematic review. (The tool is available for your personal use by downloading it from < http://sph.nhs.uk/sph-files/casp-appraisal-tools/S.Reviews%20Appraisal%20Tool.pdf/view>.) This chapter will take each CASP question and present the pertinent issues. You may need to refer to previous chapters to cover methodological and statistical issues in greater detail. Each question can be answered by one of three options: Yes, Can't Tell, or No.

Screening Questions

The first two questions are designed for the clinician to determine if detailed reading of the review will be useful in answering the clinician's question.

• 1. Did the review ask a clearly focused question?

  Consider if the question is "focused" in terms of:

  • The population studied

  • The intervention given or exposure

  • The outcomes considered

• 2. Did the review include the right type of study?

  Consider if the included studies:

  • Address the review's question

  • Have an appropriate design

In Chapter 10 we discussed the need for the research we seek to be focused on the four parts of the clinical question: Problem or Population, Intervention, Contrast intervention, and Outcome. We also discussed the importance of having the right Design type to match the question of interest (PICO + D). These two screening questions (1 and 2 above) can help clinicians determine that the review includes the right set of literature using the right designs.

One of the main issues to be considered before applying the results from a systematic review is whether or not the results are likely to be valid.⁹ This issue will be revisited throughout this checklist. But, the first consideration of validity is the review's face validity; that is, does the topic appear to include the elements that make it relevant to the clinician, for the particular question?

Clearly Focused Question

Population: In the description of subjects, one must check that all the relevant details of the patients are provided, such as age, sex, diagnosis, and time since onset of problem. Also, these factors must be sufficiently narrow that the results are meaningful. For example, a review that looks at fracture healing rates should not combine results from teenagers through the elderly, unless they are divided into subgroups, given the inherent differences between the age groups. Physical therapists often find that a question is both clearly focused and well constructed but not entirely relevant to their patient. In this case, it is worth checking the review to see if a subgroup analysis has been carried out, which might be more closely related to a specific patient.

Intervention: A poor description of the intervention is frequently the downfall of a potentially good study. In many areas of physical therapy, treatment philosophies are compared without adequate description of the actual intervention. There is therefore no way of ensuring that treatments were equal, and they are certainly not reproducible. You may be surprised to find how few studies are included in a review of an area in which there has been much published. The application of rigorous standards for inclusion often results in a large cull of inadequate papers.

Contrast Intervention: Because of the nature of systematic reviews, they will not include contrast interventions, although they will discuss the interventions provided to any control groups.

Outcome: Because the decision to apply results has to be based on the changes in outcomes that the intervention produced, having a clear statement of outcomes is essential. It is also the only way that results can be compared across multiple studies.

Right Type of Studies

Applicability to Question: It can be frustrating at times to find a review of a related topic but one that does not answer your specific question. This is often the case as, through necessity, the question narrows the population, outcome, and intervention under consideration. For example, intervention studies in stroke often exclude so many comorbidities that the clinician has very few patients who match the sample.

An explanation of the importance of the information covered and how it fits into the context of previous work is a good sign. This should be followed by an explicit statement of the criteria used to choose the studies to be included in the review to protect against selection bias.

Design: defining "appropriate" with regard to the study design is left to the clinician. Many consider the gold standard to be a randomized controlled trial (RCT), but this level of rigor has often not yet been carried out in physical therapy. And so the reviewers may have decided to accept studies that fall further down the Hierarchy of Evidence scale, such as an experimental study without randomization. In Chapter 12 we discussed several reasons, with their advantages and disadvantages, for choosing other research designs besides RCT (see Chapter 12). Of equal importance, and sometimes overlooked, is the fact that an RCT may not be the best design for a particular question. This may occur when the research question cannot be answered by a comparison at the time, for example, when there is no gold standard of treatment against which to compare. In this instance, a case series would be useful (see Chapter 12). Finally, designs other than RCTs may have been considered because an RCT is unethical. This is often the case in physical therapy, where the optimal design would be to compare one intervention to no treatment.

Insisting upon carrying out reviews using only RCTs limits improvements to therapy practice. A compromise sometimes employed is the analysis of subgroups of information based on their design type. This helps to present the full spectrum of information available at the time. A good review also sets out the literature limitations in its introduction and explains its reasons for the choice of study designs included, thereby further educating the reader.

If the answers to the questions above are "Can't Tell" or "No," then the CASP authors suggest serious thought be given to whether it is worth continuing to read the review.

Detailed Questions: The Process of Review

The remaining questions focus on the quality of the review and also help the clinician determine if and how to apply the results of the review to their patients.

• 3. Did the reviewers try to identify all relevant studies?

  Consider:

  • Which bibliographic databases were used

  • If there was follow-up from reference lists

  • If there was personal contact with experts

  • If the reviewers searched for unpublished studies

  • If the reviewers searched for non-English-language studies

Identification of Studies: Including only some data available on a topic introduces systematic errors and threatens the validity of the results. A number of data sources are difficult and time-consuming to find, and yet, the effort needs to be made. Egger and Davey-Smith¹⁰ found that among published studies, those with significant results are more likely to get published in English, more likely to be cited, and more likely to be published repeatedly, leading to English-language bias, citation bias, and multiple-publication bias. This likelihood that studies with significant results (the results show that the intervention had a significant positive, or sometimes negative, effect) are more likely to be submitted and accepted for publication than those that show no results is termed publication bias. Systematic reviews that do not consider difficult-to-access information may be prone to these biases. It is therefore important to note how a review has explicitly considered each of the following issues.

Databases: A sign of a high-quality review is a thorough search of the information available to answer the research question. This process should be sufficiently described to allow replication. The process begins with complete coverage of the databases used to find articles. The databases used depend on the topic, but a standard procedure within physical therapy would be to include a search of the databases in Box 15.1, all of which are discussed in greater detail in Chapter 11.

Use of Reference Lists: Researchers should explicitly state the measures they have taken to ensure thorough hand searching. At the very least, the Cochrane Collaboration should have been checked. Examples of good practice include searching through the reference lists from identified studies, systematic use of the "Related Articles" function within the databases searched, and a check of key journals over the past year (as it may take this long for them to appear in a database).

Unpublished and Gray Literature: A thorough review will take extra steps and search sources such as conference presentations, abstracts, posters, technical reports, theses, and unpublished data. These informally published pieces of work are known as gray literature. By including gray literature, a systematic review is better able to overcome publication bias. Hopewell et al¹¹ investigated the impact of gray literature in meta-analyses of RCTs of health care intervention. Of five studies that compared the effect of the inclusion and exclusion of gray literature, all showed that published studies demonstrated a greater treatment effect than did the gray trials—as much as 9% when the data of three of these studies were combined. Burdett et al¹² looked at the combined effects of gray literature and non-English journals and found that they were less favorable, therefore moving the estimated treatment effect toward a null result.

A step beyond searching the gray literature is contacting researchers for unpublished data or ongoing work through discussion groups and contacting experts in the field. Davey-Smith and Egger¹³ remind us of the danger in treating unpublished and published material in the same way. They suggest that unpublished data be heavily scrutinized, as it has not undergone peer review, and consideration be given to the possibility that its source may have a vested interest, for example, a manufacturer. Mosteller and Colditz¹⁴ go so far as to recommend that data from unpublished work should be considered separately from peer-reviewed data. Therefore, a discriminating reader should:

• Look to see that a systematic review has searched the gray literature and included it where appropriate.

• Consider the amount and quality of the unpublished and gray literature presented. An overreliance on it would raise questions about the quality of the review.

Inclusion of Foreign-Language Information: To make the review comprehensive and accurate, evidence suggests that all articles in all languages should be included for consideration, to avoid publication bias secondary to the selective availability of data. Egger and Davey-Smith¹⁴ suggest that "positive" findings may be more likely to be published in an international journal in English, whereas "negative" findings may be more likely to appear in a national journal in the original language. Other researchers have found that the exclusion of trials published in languages other than English did not affect the meta-analyses and therefore did not bias estimates of intervention effectiveness (Juni et al¹⁵ and Moher et al¹⁶). The conclusion seems to be that the inclusion of trials published in languages other than English is a sign of a better quality review because the researchers have been thorough. However, evidence is insufficient at this stage to count a lack of inclusion against them. Reviews should be explicit about whether or not they used English-only publications.

• 4. Did the reviewers assess the quality of all relevant studies?

  Consider:

  • If a clear, predetermined strategy was used to determine which studies were included. Look for a scoring system and more than one assessor.

Quality of the Relevant Studies: It is essential that the reviewers choose only studies that meet certain standards of quality. The inclusion of poor-quality studies is likely to distort the results of a meta-analysis and the conclusions of the review and therefore inject bias into the review because of bias in the individual studies. As was discussed in Chapter 12, there are several types of bias that can affect internal validity, among them selection, performance, detection, and attrition bias. The quality of research is related to the means the researchers used to reduce each of these biases.

A Scoring System: A number of scales have been developed in an attempt to help the reader to assess the quality of systematic reviews and meta-analyses. Despite the large number of available scales, no single one has emerged as a clear favorite. We chose this CASP checklist because it considers the factors that are absolutely necessary to prevent types of bias and provides the best combination of effectiveness and efficiency.

Use of More Than One Assessor: Because of the subjective nature involved in choosing to include or exclude a study from a review, it is good practice for at least two assessors to have been involved in the decision making. Check to see that this has been stated, as well as a procedure for resolution of disagreements.

• 5. If the results of the studies have been combined, was it reasonable to do so?

  Consider whether:

  • The results of each study are clearly displayed

  • The results were similar from study to study (look for tests of heterogeneity)

  • The reasons for any variations in results are discussed

• 6. How are the results presented and what is the main result?

  Consider:

  • How the results are expressed (e.g., odds ratio, relative risk, etc.)

  • How large the result is and how meaningful it is

  • How you would sum up the bottom-line results of the review in one sentence

• 7. How precise are these results?

  Consider:

  • If a confidence interval were reported, would your decision about whether or not to use this intervention be the same at the upper confidence limit as at the lower confidence limit?

  • If a p-value is reported where confidence intervals are unavailable

Integration of the Results of Multiple Studies

As we have said, often the research found on a given topic may be so disparate that the results of each study cannot be combined in any way and the reviewers must instead provide narrative summaries to help guide the clinician. But, sometimes, the research does have enough similarities in design, subjects, methods, and outcome measures that the results can be combined. This is especially helpful when the studies either have differing results or when the similar studies may have been too small to be able to glean useful effects. Finding that the research can be combined means that clinicians will have access to more powerful data to help in their decision making. Questions 5 through 7 all deal with this issue by asking if reviewers made the right decision in combining and if they combined appropriately. This combination is done through a series of statistical procedures known collectively as a meta-analysis. Meta-analyses are often, but not always, presented as part of a narrative review of the research studies. This means that clinicians can see the quantitative summary of the results as well as find synopses of each study. These three questions all deal with the quality of the methods used by the reviewers in combing results and will be discussed together.

Reasonableness of Combination: Clearly, the initial question a clinician must ask in reading the review is whether it was reasonable to combine the studies. There are many aspects to considering this question, as this is really an example of the old phrase "garbage in, garbage out" being true. First look at the actual articles. Does it make sense clinically to you that these articles are grouped together? Do these look like "good" articles to you based on the principles in Chapter 12?

Funnel Plots for Meta-Analysis: Earlier we discussed the problem of publication bias, that is, the possibility that the studies available for review overrepresent work that showed positive, large effect sizes, and underrepresent work that had non-statistically-significant results. Many authors of reviews use a funnel plot as the first part of their quantitative analysis to help screen for this problem. A typical funnel plot will have studies arranged by size on the vertical axis and by variance from the pooled estimate of effect size (often such measures as relative risk or odds ratio) of all the studies on the horizontal axis.^17,18 Studies with small samples tend to have greater variation in treatment effect due simply to chance, whereas the opposite is true for larger studies. Therefore when the available studies represent a range of results around the pooled estimate of effect, the cluster diagram will resemble an inverted funnel. An asymmetric funnel indicates a relationship between treatment effect and study size. If the reviewers find an asymmetrical result in the funnel plot, they have several options. One is to not proceed with further quantity analysis, but to do qualitative analysis, with a warning to the reader about the potential bias; another is to seek the gray literature discussed above and recalculate the funnel plot, since that may well result in greater symmetry; and a third is to determine reasonable causes of the asymmetry other than bias and proceed with the analysis. The important thing for the clinician is that the reviewer address the issue in a reasonable way that makes clinical sense.

Heterogeneity: While this type of assessment is a good start, using statistical analyses that take into account the variations in similar studies can provide more definitive data and removes any personal biases we may have about the studies. It is important to determine if the studies are similar enough to be grouped together. This question deals with the issue of homogeneity versus heterogeneity. We mentioned these concepts in Chapter 12 in reference to the variability in the subjects in a study. Here, we are asking how different were the populations, interventions, and outcomes studied? Are their results comparable? When you see a variety of effects of the intervention upon an outcome, it is important to be able to determine if the results are actually related to the intervention itself, and not the result of either clinical heterogeneity, that is, differences in populations, interventions, and outcomes themselves, or methodological heterogeneity, where there are differences in study design or quality.

Heterogeneity is measured statistically as part of the meta-analysis, as we discuss below. But the reviewers need to do several steps initially to make sure that it even makes sense to do the statistical analyses. The reviewer should identify issues that had good potential for causing heterogeneity at the beginning of the studies. A good review will show or state that a table of studies was developed to explore whether or not there are any unsuspected relationships that might have affected the size of the effect. If possible relationships are found, then subgroups should be established prior to even beginning the statistical analyses.

The results of each study should be displayed clearly and in a manner that allows easy comparison across studies. The format used by the Cochrane group is a good example. Here is an example taken from the systematic review, "Supportive devices for preventing and treating subluxation of the shoulder after stroke."¹⁹ One can easily see in the summaries (Fig. 15-2) comparing Ancliffe²⁰ and Griffin²¹ that the interventions and the outcomes are identical. However, as is often the case, there are some variations. The methods vary slightly (Griffin's study includes a placebo group), and the participants are also different (Ancliffe's subjects have more acute stroke, while Griffin's had a higher male-to-female ratio, a larger number of subjects, and a greater proportion of subjects with right hemiplegia).

If you find in a systematic review that there are apparent differences between the populations, interventions, or outcomes which you feel are sufficient to skew the results, this may be a sign of a poorer-quality review, particularly if the reviewer does not recognize the variability and discuss the implications, including the potential for future research.

Most first-time review readers, especially if they are reading in an area outside their expertise, will not feel capable of distinguishing between acceptable and unacceptable degrees of heterogeneity. In the example above, it is difficult to tell from just visual inspection whether or not the population variations prevent the groups from being comparable. In addition, because many studies are relatively small, it may be difficult to assess their results simply by examining the numerical results of individual studies (see Chapter 12). The use of a meta-analysis, combining the data from individual studies, allows the clinician to get a clearer picture of the intervention effects than does examining studies one at a time.

Presentation of Results in Forest Plots: A forest plot is part of a meta-analysis and is a way to visually present the results of the studies in the review. It provides information about the magnitude, direction, and precision of the individual effects of all the studies in one place. Forest plots can provide an amazing array of information to clinicians, once all parts of it become clear. See Figure 15-3 for an analysis of a forest plot from a review of progressive resistance training.²²

As can be seen from the figure, the forest plot can provide all of this information in its visual display:

• The independent and dependent variable being analyzed in the plot (A)

• The number of total studies in the review being summarized in the forest plot (B)

• The authors and year of publication for each article (B)

• The results from each study (mean, standard deviation, and number of subjects in each group, (C))

• The relative weight of each study in the analysis based on its sample size (D)

• A numerical reporting of the confidence intervals for these studies (E)

• A uniform scale (e.g., standardized mean difference) to allow comparisons across the studies when a common outcome scale has not been used (E)

• Statistics (chi-square being the most common) to help determine if the level of heterogeneity is satisfactory (F)

• A plot of the relative outcomes of each study that shows:

  • By length of line, the full confidence interval

  • By size of symbol, the relative sample size, and

  • By location on each side of the centerline (no differences between groups), the direction and strength of the results (G)

• A summary line that shows the results from the integration of the studies in the review (H)

By scanning the forest plot, a clinician can immediately see the depth of literature available, the sample size and results of each study, and the individual and collective weighted results. For example, in this systematic review of progressive resistive exercise, the reviewers identified 33 studies that measured change in function (using a variety of tools) with a total of 2172 subjects. Twenty-six of these studies had results favoring progressive resistive exercise, and the weighted standardized mean difference for all the studies also favors the intervention. Because the outcome measured varied, necessitating the use of a uniform scale, we cannot identify the degree of clinical change that this represents.

Managing Heterogeneity: As we have mentioned, reviewers may decide that there are subgroups of studies based on differences in either the designs (methodological or statistical heterogeneity) or in the subjects, interventions, or outcome measures (clinical heterogeneity). Subgroup analyses can offer a solution to both of these types of heterogeneity. For example, a reviewer might divide studies into randomized controlled trials (RCTs) and nonrandomized controlled trials. In this way, analysis of the results from the RCTs will not be confounded by the potentially less reliable results from the nonrandomized studies. (See Chapter 12 for a discussion of why the results may be less reliable.) Or the reviewers may develop separate forest plots (as in our example systematic review) for studies that used different outcome measures or different subjects.

Precision of the Results: Whether or not a forest plot is provided, summary statistics are used to give readers the results of the review. Using the wrong statistics or misinterpreting the results of the statistics can result in two types of errors. It is possible to miss a clinically important result, and it is also possible to produce results that appear clinically important, but actually are not.

The most common statistics used are the odds ratio and the relative risk (sometimes expressed as a risk ratio). They are appropriately used for assessing dichotomous data, where the outcome for each participant is one of two possibilities, for example, harmed or benefited. Both of these statistics compare the likelihood of an event between two groups. The odds ratio provides the strongest mathematical result, but the risk ratio provides the more easily interpreted result.

It is important to understand when their use is appropriate and why a researcher or reviewer might choose to use one over the other. Although in everyday language we use the terms risk and odds synonymously, this is not the case in statistics. They may be used to describe the same data, but they are expressed very differently. Relative risk is often preferred because it more closely matches how we think. When the risk is 0.25, or 25%, we know that 25 people out of 100 will be affected. However, some designs prevent its use, particularly the case-control design, which selects research subjects on the basis of the outcome rather than on the exposure.

When reading a systematic review, it is important that you know which of these statistics is used. As both are graphed on a scale where the center value is 1, it is not immediately obvious which statistic is being displayed. A common mistake is the misinterpretation of odds ratios as risk ratios, causing an overestimation of the benefits and harms of an intervention. Deeks and Altman²³ suggest that use of both statistics provides the most useful information—the odds ratio could be used for mathematical strength while the risk ratio could then be presented for ease of interpretation. This is not a necessary requirement for a high-quality review, but it is a mark of deeper consideration by the review authors if it is provided.

It is also important to recognize the differences between the two for clinical relevance. This sounds obvious, but whereas a risk of 0.5 (50% chance of an event) is equivalent to odds of 1 (for every 2 people, 1 will experience an event) is easy to see, a risk of 0.95 being equivalent to odds of 19 is not.²⁴ Interestingly, McGettigan et al²⁵ found that the choice of the statistic used to present a result affected doctors' opinions or intended practices. In a systematic review of 12 RCTs, 8 compared the framing of a result as a relative risk reduction, absolute risk reduction, or number needed to treat. The authors found that the risk ratio (relative risk reduction) was viewed most positively.

Result Size and Its Meaning: Statistical significance tells us nothing about the size of the effect or its clinical relevance. For effect size, one must go back to the forest plots of relative risk and odds ratios and the summary effects of each. Be sure to review the presentation of these values, paying special attention to the midline of "no effect." The further the point estimate of effect is from the line of no effect, the greater the magnitude of the effect. But how do you determine the absolute magnitude of the effects and how much should this influence clinical practice? Are the results relevant to all your patients within the same population parameters, or only to a subset? These are questions you must answer from a clinical perspective.

Another important aspect of interpreting the results of a meta-analysis is consideration of the summary effect produced by the statistical model used. There are two: the fixed effect model and the random effects model. One of these two statistical methods will be used to analyze the pooled data. Note that our example uses a fixed effect. Because of the nature of these two analyses, a random effects model may overemphasize the importance of smaller studies with poorer quality. A fixed effect model gives preferential weighting to larger studies, thereby producing a more narrow summary effect. This may incorrectly give the impression of greater precision of the effect. Although the type of model used will not help you to determine a good-quality review from a poor one, bearing in mind the caveats of each will make you a more discerning reader, better able to interpret and apply the results appropriately.

Meta-analyses must be done carefully and appropriately to be useful to clinicians. A perceptive reader will look for the issues of bias and clinical and methodological heterogeneity, as previously discussed, as well as carefully consider the clinical meaning of the numerical values that represent the study results.

Detailed Questions: The Bottom-Line Results of the Review

Ultimately, the bottom line presented by the reviewers should explicitly answer the initial research question. This does not mean that the answer definitively shows harm or benefit, but that it provides an accurate portrayal of the data and their meaning. The bottom line in physical therapy is too often that the effectiveness of a treatment cannot be determined with the data available at the present time. Particularly in areas where research has been investigating the effectiveness of a practice that has been key to physical therapy, this may come as a surprise. Cautious interpretation may lead to suggestions for clinical practice in this case, but the reviewer should not make claims beyond the results of the current research. Updates of the review may find stronger relationships or effects in the future, but then again, perhaps not!

• 8. Can the results be applied to the local population?

  Consider whether:

  • The population sample covered by the review could be different from your population in ways that would produce different results

  • Your local setting differs much from that of the review

  • You can provide the same intervention in your setting

Applicability to Your Patient: The first step is to go back to the initial research question and be sure that it is answered. Sometimes the literature is not sufficiently robust at the time to provide a definitive answer, or the availability of literature is so limited that answers of significant power do not apply to your patient. The second step is to determine if the results are statistically significant, and if so, are they clinically important/relevant? And finally, you must determine if the results are pertinent to your specific patient(s). Sometimes a "bottom-line" that considers clinical effectiveness, as well as feasibility and meaningfulness of the intervention, will be included in the review. This is handy but still requires your judgment for applicability to your patients. Points to consider in making that judgment include:

• Are the subjects and the intervention sufficiently described to allow comparison to your patients?

• Was the intervention effective in all patient subgroups?

• Were the outcomes used in the studies relevant to your patients?

These would appear to be obvious points, but there may be subtle but important differences between your patient population and those considered in the review. For example, many studies consider the effect of muscle strength training upon the "elderly" but do not subdivide age groups. If you are treating patients primarily over 80, then a review that defines the subjects as "over 65 years" is not likely to be adequately specific, and therefore applicable, to your patients. Particularly if you are unable to find a review whose patients match your own, it is imperative that you know enough about the features that do not match to be able to gauge their relative importance. So in the previous example, you would need to investigate whether the age difference might be an important factor, and in this case the evidence would suggest that it is.

Often the issue is that your patient is not "ideal," that is, he or she would not have met the inclusion criteria for the studies due to comorbidities. In the example above, the recommendation is not clear-cut. An understanding of the pathophysiology of the disease and the proposed mechanism of action of the treatment is necessary to make a decision whether or not to apply the results to your patient.

Systematic reviews are not just carried out to ascertain treatment effectiveness. Feasibility of treatment is also an important question, so the type of study design is a consideration. Unfortunately, many of these kinds of practical clinical questions will not have been addressed in the literature, especially if they do not lend themselves to an RCT format. An example would be the decision making for cost-effectiveness.

Number Needed to Treat: If your question involves cost or time, going back to the number needed to treat (NNT, the inverse of the absolute risk reduction) may give guidance. Knowing how many people must be treated before you see benefit may help you to decide if the intervention is warranted. It is up to the clinician to decide the level of the NNT before it is worth changing practice. Some authors suggest an NNT of three or less indicates adopting the treatment.²⁶ However, that depends on the value of the outcome. If the outcome is the prevention of serious pain, then a much larger NNT would still be considered worthwhile. If the value of the outcome is not so clear, then a consideration of the number needed to harm (NNH) may help in the decision making (see Chapter 12 for further detail). As the NNT increases, then patient and clinic resources become more of a consideration, as more patients must receive the treatment before one new patient improves. In fact, a treatment with little resource requirements might seem preferable over one with a better NNT, if this improves overall service provision.

• 9. Were all important outcomes considered?

  Consider outcomes from the point of view of the:

  • Individual

  • Policy makers and professionals

  • Family/careers

  • Wider community

The first aspect to consider is whether or not the outcome measures used in the studies are the same as, or relevant to, those you use in the clinic. In many areas of physical therapy, this is a big issue, as often outcome measures of activity are used when clinicians are most interested in the effects of treatment upon functional and structural integrity. For example, you are working with a stroke patient in an acute rehabilitation setting and you are interested in increasing the movement available in the affected leg to improve his safety in walking. You come across a systematic review of an intervention that uses the Functional Independence Measure (FIM) as the outcome measure, and it shows that there is a significant decrease in need for assistance while walking. But this does not tell you what aspect of movement improved to allow greater independence—was it flexibility, strength, balance? In an ideal world, the research and clinical measures would be the same; when they are not, you will need to carry out your own cost/benefit analysis and use your clinical judgment.

Another consideration will be whether the outcomes are meaningful to the experiences, values, and beliefs of your patients, as well as whether there are aspects of the intervention that are likely to upset a patient and negatively affect the outcome. To use the stroke patient example—he may feel too easily fatigued to agree to an intensive treadmill training regimen, which has been shown to be effective in similar patients but is upsetting to him.

• 10. Should policy or practice change as a result of the evidence contained in this review?

  Consider:

  • Whether any benefit reported outweighs any harm and/or cost. If this information is not reported, can it be filled in from elsewhere?

In addition to considering the positive value added by implementing such policy change, as we've discussed in preceding questions, it is essential to balance this with the potential harm from implementing the change. Recent research often includes information on mortality, morbidity, quality of life, cost-effectiveness, and patient satisfaction.²⁷ "Harm" is rarely measured as such; more often, data is collected on causes of dropout or side effects. This makes a benefit/harm analysis lopsided. Therefore, the best you might do is to consider benefit and cost, which were covered with NNT analysis in Question 8.

Synthesis Conclusion

Systematic reviews can provide a quick answer to your clinical questions. Despite the best attempts to make data quantifiable and systematic reviews objective, there are many opportunities throughout the process to introduce subjectivity. Regardless of the number of people involved (and sometimes because of the number of people involved), each time a decision is made about methodological quality or the presence of unacceptable heterogeneity, bias may be introduced. Ultimately, the clinicians will impose their judgment upon the quality of a review by using the guidelines above. It is hoped that by using the CASP guidelines these decision will be better informed.

SYNOPSES OF SYNTHESES

Systematic reviews are very valuable because they synthesize so much information in one place. But this very comprehensiveness means that they there are often quite long and somewhat daunting for a clinician to use. Cochrane reviews now include a "plain language" summary of the full review, but even these summaries can reach over 50 pages in length. In response to this, synopses of systematic reviews are being created, just as they have been created for individual studies.

The largest source of synopses of systematic reviews is the Database of Abstracts of Reviews of effects (DARE). DARE was created and is maintained by the Centre for Reviews and Dissemination of the University of York (United Kingdom). Potential reviews of interventions are identified by searching leading journals, bibliographic databases, gray literature, and selected Web sites. The reviews are then assessed by two researchers who determine inclusion by assessing the reviews against these five criteria:

1. Were inclusion/exclusion criteria reported?

2. Was the search adequate?

3. Were the included studies synthesized?

4. Was the validity of the included studies assessed?

5. Are sufficient details about the individual included studies presented?

To be included, the review must meet at least four of these criteria, including 1 to 3, which are required. Each abstract includes review methods, results, and conclusions and provides a critical assessment of the systematic review. Each abstract is shared with the systematic review authors for correction of factual errors before publication. (See Fig. 15-4.) DARE can be found at many databases, such as Ovid or APTA's Open Door.

Synopses of syntheses can be found from many other sources as well; for example, they are published in evidence based journals, such as a synopsis of a systematic review of exercise to improve depressive symptoms that was published in Mental Health.²⁸ Physical Therapy has begun publication of a series known as Linking Evidence and Practice (LEAP). LEAP articles are based on actual patient scenarios and provide useful clinical information from systematic reviews available from the Cochrane Collaborative and other reliable sources. Their goal is to improve the application of information from systematic review to daily physical therapy practice.^29,30

Just as the number of synopses of individual studies has increased, there will be an increase in the number of synopses of syntheses. These synopses can provide information to clinicians in a way that is clinically useful. They can also be created by clinicians as a useful way to share the literature among themselves at events such as journal clubs and as part of professional development programs.³¹ There will also be an increase in articles that bring together the results of several systematic reviews.³²

CONCLUSION

Syntheses and their synopses can serve the clinician by gathering the results of many individual studies in one source. As with all sources of evidence, the clinician must assess the quality of systematic reviews before implementing their recommendations. This assessment includes an examination of the technical components of the review but also involves a clinical judgment about the applicability of the findings to specific patient questions.

SELF-ASSESSMENT

1. Find a systematic review related to the value of aerobic exercise for improvement of cardiovascular status in patients with cardiovascular disease. Using the CASP questions, determine the quality of the systematic review.

2. Does the systematic review give you enough information to determine if the results apply to patients like Mr. Ketterman? Does it give you enough information to design a specific intervention?

3. Identify three or four systematic reviews on topics of interest to you that contain forest plots. Examine each of the elements of the forest plots and then write a few paragraphs that translate these points into a narrative. Do you prefer the graphic presentation of the forest plot or the narrative form?

CONTINUED LEARNING

• Arrange a journal club session in your practice where each participant presents systematic reviews.

• Writing a systematic review is a challenging activity but one that can help clinicians bring together a body of literature to answer important clinical questions. Writing a review is also an excellent way to gain mastery of search and assessment skills. The best guide to writing a systematic review is the Cochrane Collaboration's Handbook, which can be found at www.cochrane.org/training/cochrane-handbook.

REFERENCES

INTRODUCTION

As we have seen, studies (Chapters 12 and 13), synopses of studies (Chapter 14), and syntheses and synopses of syntheses (Chapter 15) all offer clinicians evidence that can help guide clinical decision making in patient-centered physical therapy care. But we also have seen that in almost all cases, this evidence is focused on one specific aspect of that care, be it diagnostic and prognostic tests or interventions. As clinicians, we want to understand the evidence about the full range of care we provide to patients. The efficiency of having the entire pattern of care reviewed becomes even more obvious when we are focusing on a patient population that represents a large part of our practice. Summaries and systems, the final two elements of Haynes's 6S pyramid (Fig. 16-1) offer us just such information.^1,2 They integrate a range of evidence across available studies and syntheses to provide guidance about management of a particular health problem.¹

SUMMARIES

Practice Guidelines

Perhaps the most common form of summary is an evidence-based clinical practice guideline.^1,2 Because the term guideline is used in so many different contexts, we will take a step back to come to a common definition before proceeding with resources and examples. Eddy³ has described in detail the different kinds of policy statements available to guide care, as is shown in Figure 16-2. Along the width (x-axis) of the cube is the intended use of any policy, ranging from providing advice, through approvals for payment, to approvals for allowing practice, such as credentialing by facilities and insurance companies. Most of us have experienced policies governing each of these functions. In this chapter we are primarily interested in the use of policies based on evidence to provide advice, although it is certainly possible that payers and employers may develop policies based on evidence.

Along the depth (z-axis) of the cube is the type of guidance intended. Will the policy dictate a particular path to which the practitioner adheres, or will it set limits within which the practitioner chooses among various options? Again, we are familiar with these two options. Many facilities have developed clinical paths to help provide optimal care to groups of patients.⁴ The Guide to Physical Therapist Practice is a leading example of a boundary-type policy document. The Guide includes over 30 practice patterns that outline acceptable options for examination, intervention, and outcome measurement for specific patient populations.⁵ It is up to the clinician to choose from among these options as appropriate for the individual patient.

Along the height (y-axis) of the cube is the intended flexibility range of the policy statement. The most rigorous type of practice policy is the standard. When we are able to reach a high level of certainty about what ought to happen, then the policy can be called a standard and the practitioner can be held accountable for meeting the standard. Penalties for lack of conformance to the standard can include loss of licensure, malpractice litigation, denial of payment, or loss of employment. Obviously, such high-stake penalties mean that the standard must be supported by a very high level of evidence that it is, indeed, the correct standard. Eddy³ states that development and adoption of standards should be based not only the scientific evidence but also on the nearly unanimous clinical wisdom of practitioners and nearly unanimous agreement among patients about the desirability (or undesirability) of the outcome of the care adopted in the standard. At this writing, there are no published standards that are specific to physical therapy. Of course, there are standards of care that cross disciplines, such as those related to infection control, which are applicable to physical therapists. But the total number of things in health care about which relatively unanimous agreement can be reached is very small.

At the next level on this axis are guidelines. When we have enough information to make decisions about the appropriate use of a test or an intervention, and a majority of clinicians and patients would agree about the desirability (or undesirability) of the outcome of the care, then they can be considered guidelines. Guidelines serve as recommendations about what is preferred in practice. Clinicians may well choose to do something different, but that choice should be based on factors such as a patient's health variability and preferences. We'll discuss later the growing body of policy statements that are offered specifically to physical therapists as clinical guidelines, as well as the many guidelines that are cross-disciplinary in nature. Despite the growing number of guidelines, they still cover only a small fraction of practice.

At the third level on this axis is the category of option. A policy statement should be considered an option if there is some uncertainty about outcomes of the particular test or intervention, or if clinicians' and patients' preferences about the outcomes are unknown, unclear, or equivocal. The practice patterns in the Guide to Physical Therapist Practice⁵ represent a good example of policy documents that offer clinicians options for selecting examination tools and interventions for identified patient groups.

Returning now to Haynes's model, the clinical guidelines that we will be discussing in this chapter are most closely aligned with those policy recommendations that are termed guidelines by Eddy. They may be either boundary or pathway documents, and they serve a variety of purposes, ranging from advice to practitioners to coverage and payment decisions by insurers. These types of guidelines are produced by many sources. They can come from professional or health care delivery organizations and governmental agencies. What they have in common is that they are based on evidence from the literature, as assessed by a group of experts, and offer recommended patterns of care incorporating several, if not all aspects of care for a particular group of patients.

Sources for Guidelines

Many resources are available for identifying clinical guidelines. The source with the largest number of guidelines is the National Guideline Clearinghouse (NGC),⁶ which is a service of the Agency for Healthcare Research and Quality (AHRQ)⁷ of the U.S. Department of Health and Human Services. The NGC may be accessed by anyone at www.guideline.gov. It contains much useful information about guidelines, including:

• Structured abstracts (summaries) about the guideline and its development.

• Links to full-text guidelines, where available, and/or ordering information for print copies.

• A Guideline Comparison utility that gives users the ability to generate side-by-side comparisons for any combination of two or more guidelines.

• Unique guideline comparisons, called Guideline Syntheses, that compare guidelines covering similar topics, highlighting areas of similarity and difference. These syntheses often compare guidelines developed in different countries, providing insight into international health practices.

• An Annotated Bibliography database where users can search for publications about guideline development and implementation.

• An Expert Commentary feature written or reviewed by members of the NGC Editorial Board.

In order for a guideline to be accepted for publication at NGC it must meet all of these inclusion criteria:

• The clinical practice guideline contains systematically developed statements with recommendations, strategies, or information that assist health care practitioners and patients to make decisions about appropriate health care for specific clinical circumstances.

• The clinical practice guideline was produced under the auspices of medical specialty associations; relevant professional societies, public or private organizations, or government agencies at the federal, state, or local level; or health care organizations or plans.

• Corroborating documentation can be produced and verified that a systematic literature search and review of existing scientific evidence published in peer-reviewed journals was performed during the guideline development.

• The full-text guideline is available upon request in print or electronic format (for free or for a fee), in the English language.

• Documented evidence can be produced or verified that the guideline was developed, reviewed, or revised within the last 5 years.

It is easy to find guidelines at NGC simply by entering free text into the search box. For example, a search on knee pain produced 59 different guidelines; on stroke, 350 guidelines; on aerobic exercise, 71 guidelines.

Guidelines in Physical Therapy

Another source of guidelines, one which is much more specific to physical therapy, is the Physiotherapy Evidence Database (PEDro)⁸ (see Chapter 14). It is also relatively easy to identify guidelines in PEDro simply by specifying practice guideline in the search strategy and then selecting the topics from drop-down menu choices. For example, a search for practice guidelines on the same topics as the NGC search meant selecting "lower leg or knee" from a drop-down menu, which identified 28 guidelines, selecting "neurofacilitation" and "motor coordination," which identified 5 guidelines, and selecting "reduced exercise tolerance" and "cardiothoracics," which indentified 160 guidelines.

Several professional organizations have begun development of clinical guidelines as well. The Orthopedic Section of the American Physical Therapy Association has started a project to develop evidence-based guidelines that use the International classification of Functioning, Disability, and Health (ICF) to describe categories of patients with various musculoskeletal conditions.^9,10 These guidelines are designed to provide recommendation on all aspects of care for each condition, including examination, diagnostic classification, prognosis, intervention, and assessment of outcomes. The strength of the recommendation is based on the assessment by the authors of existing peer-reviewed literature. Figure 16-3a, b, and c shows a summary of the recommendations for the condition of heel pain–plantar fasciitis.

The American Physical Therapy Association has a major project to develop clinical guidelines for the practice of physical therapy. Its first steps are to identify all currently available relevant documents and to define the processes to be used to develop new clinical practice guidelines. Further information on the progress of this activity will be available at www.apta.org and at the FA Davis Web site for this text, as it becomes available.

Other Sources of Guidelines

In addition to these sources, many other organizations have produced guidelines. Some are for the purpose of guiding payment policy, such as those issued by insurance companies. Others are promulgated by delivery organizations, such as large corporate practices or multi-hospital systems to provide consistency in practice. Sometimes these guidelines are then made available for sale to the public.

Assessing the Quality of Guidelines

With this many sources, it is important for clinicians to assess the quality and applicability of the evidence before implementing guidelines in their own practices. When assessing guidelines, not only is the quality of the evidence important but it is also necessary to understand the level of (un)certainty of the recommendation being made. The strength of the recommendation is based on several factors in addition to the strength of evidence, including recognition of costs, potential dangers, etc. A system for assessing guidelines that takes into account both of these features has been developed and is in wide use in the EBP community, including the World Health Organization and the Cochrane Collaborative. This system is called GRADE (Grading of Recommendations Assessment, Development and Evaluation).^{11,12,13,14,15,16} The GRADE system first rates the quality of evidence used to support the guideline using the following rating scheme:

• High quality—Further research is very unlikely to change our confidence in the estimate of effect.

• Moderate quality—Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate.

• Low quality—Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate.

• Very low quality—Any estimate of effect is very uncertain.

The system then looks at these four factors in determining if the strength of the recommendation was appropriate:

• Quality of evidence

• Uncertainty about the balance between desirable and undesirable effects

• Uncertainty or variability in values and preferences

• Uncertainty about whether the intervention represents a wise use of resources

The system is intended to be used by developers of guidelines to alert the potential users to their quality. So one way to help clinicians be more confident about guideline use is to determine if the guideline developers have used the GRADE system. Guideline developers also use other, similar systems, such as the one illustrated in the Orthopedics Section Guidelines (see Fig. 16-3). If they do not report any such system, clinicians could still attempt to use these suggested factors in determining for themselves if the guidelines should be implemented in their setting.