Chapter 5: Reliability of Measurements

The usefulness of measurement in clinical research and decision making depends on the extent to which clinicians can rely on data as accurate and meaningful indicators of a behavior or attribute. The first prerequisite, at the heart of measurement, is reliability, or the extent to which a measurement is consistent and free from error. Reliability can be conceptualized as reproducibility or dependability. If a patient's behavior is reliable, we can expect consistent responses under given conditions. A reliable examiner is one who will be able to measure repeated outcomes with consistent scores. Similarly, a reliable instrument is one that will perform with predictable consistency under set conditions. Reliability is fundamental to all aspects of measurement, because without it we cannot have confidence in the data we collect, nor can we draw rational conclusions from those data.

The second prerequisite is validity, which assures that a test is measuring what it is intended to measure. Validity is necessary for drawing inferences from data, and determining how the results of a test can be used. Both reliability and validity are essential considerations as we explore ways in which measurement is used in both clinical practice and research. We will address issues of validity in depth in the next chapter.

The purpose of this chapter is to present the conceptual basis of reliability and to describe different approaches for testing the reliability of clinical measurements. Statistical procedures for reliability testing are presented in Chapter 26.

The nature of reality is such that measurements are rarely perfectly reliable. All instruments are fallible to some extent, and all humans respond with some inconsistency. Consider the simple process of measuring an individual's height with a tape measure. If measurements are taken on three separate occasions, either by one tester or three different testers, we can expect to find some differences in results from trial to trial, even when the individual's true height has not changed. If we assume all the measurements were made using the same exact procedures and with equal concern for accuracy, then we cannot determine which, if any, of these three values is a true representation of the subject's height, that is, we do not know how much error is included in these measurements.

Theoretically, then, it is reasonable to look at any observed score (X) as a function of two components: a true score (T) and an error component (E). This relationship is summarized by the equation

This expression suggests that for any given measurement (X), a hypothetically true or fixed value exists (T), from which the observed score will differ by some unknown amount (E). The true component is the score the subject would have gotten had the measurement been taken by a perfect measuring instrument under ideal conditions. The difference between the true value and the observed value is measurement error, or "noise" that gets in the way of our finding the true score. For example, if we measure a height of 65 in., when the true height is 65.5 in., our assessment will be too short; that is, our measurement error is −0.5 in. On a second assessment, if we measure 66 in., our measurement error will be +0.5 in. In reality, we cannot calculate these error components because we do not know what the true score really is. Therefore, we must come up with some way of estimating how much of our measurement is attributable to error and how much represents an accurate reading. That estimate is reliability.

Systematic and Random Error

To understand reliability, we must distinguish between two types of measurement errors. Systematic errors are predictable errors of measurement. They occur in one direction, consistently overestimating or underestimating the true score. Such error is constant and biased. Therefore, if a systematic error is detected, it is usually a simple matter either to correct it by recalibrating the system or to adjust for it by adding or subtracting the appropriate constant. For example, if the end of a tape measure is incorrectly marked, so that markings actually begin 0.25 in. from the end, measurements of height will consistently record values that are too long by 0.25 in. We can correct this error by cutting off the extra length at the end of the tape or by subtracting 0.25 in. from all measurements. By definition, systematic errors are constant and, therefore, do not present a problem for reliability. Systematic errors are primarily a concern of validity, because, although they are consistent, test values are not true representations of the quantity being measured.

Random errors of measurement are due to chance and can affect a subject's score in an unpredictable way from trial to trial. They are as likely to increase the observed score as to decrease it. Random errors occur from unpredictable factors such as fatigue, inattention, mechanical inaccuracy or simple mistakes. If the patient moves slightly while his height is being measured or does not stand fully erect each time measurements are taken, scores will be inconsistent. The tester might observe markings at an angle and read them incorrectly, or the tape measure might be stretched out more on one occasion than another. Reliability focuses on the degree of random error that is present within a measurement system. As random errors diminish, the observed score moves closer to the true score, and the measurement is more reliable. The assumption is made that random error is not related to the magnitude of the true score, and that if enough measurements were taken, random errors would eventually cancel each other out, making the average score a good estimate of the true score.

Sources of Measurement Error

The development or testing of a measuring instrument typically involves specification of a protocol that maximizes the reliability of the instrument; that is, procedures are detailed to ensure consistent application and scoring. In developing such a protocol, researchers have to address known or expected sources of error that could limit the reliability of the test. Once these errors are identified, they can often be controlled or eliminated to some extent. Generally, measurement errors can be attributed to three components of the measurement system: (1) the individual taking the measurements (often called the tester or rater), (2) the measuring instrument and (3) variability of the characteristic being measured. Many sources of error can be minimized through careful planning, training, clear operational definitions and inspection of equipment. Therefore, a testing protocol should thoroughly describe the method of measurement, which must be uniformly performed across trials. Isolating and defining each element of the measure reduces the potential for error, thereby improving reliability.

Even when the anticipated sources of error are controlled, a researcher is still faced with the unpredictability of the environment and the human response as a normal and inevitable part of measurement. Many instruments, especially mechanical ones, will always be subject to some level of background noise and random fluctuation of performance. Responses of raters and subjects will be influenced by variable personal characteristics, such as motivation, cooperation, or fatigue, and environmental factors such as noise and temperature. These contributions to error may not be controllable. We assume that these factors are random and, therefore, their effect will be canceled out in the long run.

The most difficult challenge to reliability testing is faced when the response being measured is inherently unstable. For instance, if we measure blood pressure, we might expect a natural fluctuation from session to session. When a response is very unstable, no one measurement can be considered an accurate representation of it, and it is virtually impossible to estimate the reliability of the instrument used to measure it. It is important, therefore, for researchers to understand the theoretical and practical nature of response variables, so that sources of error in reliability testing can be interpreted properly.

Regression toward the Mean

When we examine the effect of measurement error on reliability, we must also consider the extremeness of observed scores; that is, very high scores may reflect substantial positive error, and very low scores may reflect substantial negative error. Cook and Campbell¹ use the example of students taking academic exams to illustrate this concept. Everyone has, at one time or another, done worse than they expected on a test, with any number of possible "excuses," such as getting a poor night's sleep, being distracted, or accidentally marking the wrong space on the answer sheet. If we think of these factors as random sources of error, then the low grade is not an accurate assessment of the student's knowledge or ability; that is, the student's true score is confounded by negative error. On a subsequent test this student's grade is likely to be higher because, all things being equal, these negative "errors" would probably not be operating to the same extent. Conversely, if a student obtained an unusually high score on the first test, we might suspect that favorable conditions were operating (positive error), such as several good guesses on questions for which the student did not really know the answer. On a second test, this student is just as likely to score lower, because these favorable conditions would not necessarily exist.

This phenomenon is called regression toward the mean. It means that extreme scores on a pretest are expected to move closer, or regress, toward the group average (the mean) on a second test; that is, the error component of an extreme pretest score is likely to be less extreme on a posttest (see Figure 5.1). Therefore, using our student example, higher test scores will decrease and lower test scores will increase, moving closer to the class average—even though the students' actual degree of knowledge does not change. The reliability of the test will have an impact on the extent to which this effect will be present. As a more reliable score contains less error, a reliable test should produce a score close to the true score. Therefore, there is less chance for regression to occur. If the tests are not reliable, the error component within each test will be large, and therefore, the chances of observing a regression effect are considerably higher.

Regression toward the mean is potentially most serious in situations where subjects are specifically assigned to groups on the basis of their extreme scores (see Box 5.1). For instance, we might be interested in the differential effect of a particular teaching technique on different levels of students. Suppose we give a class a pretest to determine their initial ability and find that the average score is 80. Then we distinguish two experimental groups on the basis of these pretest scores, one composed of those scoring above 90 and the other of those scoring below 70. According to regression theory, we would expect to see both groups respond with scores closer to 80 (the total group mean) on a second test, even if the teaching technique had no effect. The effect can be minimized, however, if we can improve the reliability of the test as a measure of the students' ability.

Reliability can be conceptually defined as an estimate of the extent to which a test score is free from error; that is, to what extent observed scores vary from true scores. As it is not possible to know the true score, the true reliability of a test can never be calculated; however, we can estimate reliability based on the statistical concept of variance, which is a measure of the variability or differences among scores within a sample. The larger the variance, the greater the dispersion of scores; the smaller the variance, the more homogeneous the scores. If we were to measure a patient's blood pressure 10 times, we do not expect the scores to be identical; that is, they will exhibit a certain amount of variance. Some of this total variance in observed scores will be the result of true differences among the scores (the patient's blood pressure actually changed), and some can be attributed to random sources of error, such as position of the arm or skill of the tester. Reliability is a measure of how much of this total variance is attributable to true differences between scores. Therefore, reliability can be expressed as a ratio of the true score variance to the total variance, or

This ratio yields a value called the reliability coefficient. From this ratio, we can see that reliability increases as the observed score approaches the true score (T + E ⇒T). With maximum reliability (zero error), this ratio will produce a coefficient of 1.00; that is, the observed score is the true score. As error increases (T + E ⇒ E), the ratio approaches zero. The reliability coefficient can range between 0.00 and 1.00, with 0.00 indicating no reliability and 1.00 indicating perfect reliability. There are actually many reliability coefficients, each applied under different design conditions and with different types of data.

What Is Acceptable Reliability?

Reliability cannot be interpreted as an all-or-none condition. It is a property of a measurement system that is attained to varying degrees. Because reliability is hardly ever perfect, reliability coefficients of 1.00 are rare. Most researchers establish limits that define "acceptable" levels of reliability. Although such limits are essentially arbitrary, as a general guideline coefficients below .50 represent poor reliability, coefficients from .50 to .75 suggest moderate reliability, and values above .75 indicate good reliability. We hasten to add, however, that these limits must be based on the precision of the measured variable and how the results of the reliability test will be applied.

We hesitate to even suggest these guidelines, as they are often inappropriately applied as standards. They are intended only as a starting point to judge acceptable standards for any specific measurement. The level of acceptable reliability must be put in context. How much error is tolerable depends on the criteria for the measurement.

For instance, we generally accept a 5 degree error for goniometric measurements. But does it matter if we are assessing range of motion of the shoulder or the distal phalanx of the finger? If we performed a 6 minute walk test, how precise must we be in measuring distance walked? If we were off by 6 inches, would that change how we used that value? But what if we were measuring leg length? A difference of millimeters could change how we manage the patient. When is it okay to be "close enough"? Wainner² uses the examples of a baseball pitch in the strike zone, or a field goal in football passing through the uprights. A certain margin of error is built into the criteria for success. How this translates into acceptable clinical measurement is not so obvious.

Researchers may be able to tolerate lower reliability for measurements that are used for description, whereas those used for decision making or diagnosis need to be higher, perhaps at least .90 to ensure valid interpretations of findings. When only one form of measurement exists for a particular variable, researchers are often faced with the choice of using a less reliable test or no test at all. The validity of the test will also make a difference (see Chapters 6 and 27), especially when using measurement for diagnosis. For some purposes, even a test with moderate reliability can add sufficient information to justify its use, especially when used in conjunction with other tests. "Acceptable reliability" is a judgment call by the researcher or clinician who understands the nature of the measured variable and whether the measurements are precise enough to be used meaningfully. In a reliability study, it is the researcher's obligation to justify the level of acceptable reliability based on the purpose of the measurement.

Correlation and Agreement

Many reliability coefficients are based on measures of correlation. Although we discuss the concept of correlation in detail in Chapter 23, it is necessary to provide a brief introduction here, to understand how reliability coefficients can be interpreted. Correlation reflects the degree of association between two sets of data, or the consistency of position within the two distributions. For example, if we were to measure height and shoe size on a sample of adult men, we would probably find a correlation between the two variables; that is, those with bigger feet tend to be taller, and those with smaller feet tend to be shorter.

Reliability can be interpreted in a similar way. For instance, if we measured height on two separate occasions, we would expect that the tallest man on Test 1 would also be the tallest on Test 2; the shortest man on Test 1 would also be measured as the shortest on Test 2. If the relative position of each subject remains the same from test to test, we would obtain a high measure of correlation. We assume that any variations in observed measurements are due to random error. If systematic errors occur, the correlation would be unaffected because each subject's relative position will not change. This means that systematic errors of measurement will not have any effect on the size of the reliability coefficient.

Consider the two sets of scores shown in Figure 5.2. In both sets the scores are directly proportional for X and Y; hence, their relationship is depicted on a straight line. The relative positions of the scores are also consistent; that is, the highest score in X is paired with the highest score in Y, and so on. The scores in graph A also show direct agreement for each pair of scores. This is not the case in graph B, where all pairs disagree. But both graphs show perfect correlation. Therefore, while correlation tells us how the scores vary together, it cannot tell us the extent of agreement between the two sets of measurements. For most research and clinical applications, however, the essence of reliability is agreement between the two tests; that is, we want to know that the actual values obtained by two measurements are the same, not just proportional to each other. For example, range of motion measurements are used to evaluate joint dysfunction on the basis of actual, not relative, limitations. We need to know the true value of the limitation, not just that one patient is more limited than another. It would not be enough to know that repeated measurements were proportionally consistent; it would be necessary to establish that repeated tests resulted in the same angular measurements. Therefore, the correlation coefficient is not effective as a measure of reliability. Because of this, statistical approaches to reliability testing should include estimates of agreement that can be used in conjunction with correlation.

Estimates of reliability vary depending on the type of reliability being analyzed. We discuss four general approaches to reliability testing: test-retest reliability, rater reliability, alternate forms reliability, and internal consistency. For each approach we will identify the most commonly used reliability coefficients. These statistical indices are described in detail in Chapter 26.

Test-Retest Reliability

One basic premise of reliability is the stability of the measuring instrument; that is, a reliable instrument will obtain the same results with repeated administrations of the test. Test-retest reliability assessment is used to establish that an instrument is capable of measuring a variable with consistency. In a test-retest study, one sample of individuals is subjected to the identical test on two separate occasions, keeping all testing conditions as constant as possible. The coefficient derived from this type of analysis is called a test-retest reliability coefficient. This estimate can be obtained for a variety of testing tools, and is generally indicative of reliability in situations where raters are not involved, such as self-report survey instruments and physical and physiological measures with mechanical or digital readouts. If the test is reliable, the subject's score should be similar on multiple trials. In terms of reliability theory, the extent to which the scores vary is interpreted as measurement error.

Because variation in measurement must be considered within the context of the total measurement system, errors may actually be attributed to many sources. Therefore, to assess the reliability of an instrument, the researcher must be able to assume stability in the response variable. Unfortunately, many variables do change over time. For example, a patient's self-assessment of pain may change between two testing sessions. We must also consider the inconsistency with which many clinical variables naturally respond over time. When responses are labile, test-retest reliability may be impossible to assess.

Test-Retest Intervals

Because the stability of a response variable is such a significant factor, the time interval between tests must be considered carefully. Intervals should be far enough apart to avoid fatigue, learning, or memory effects, but close enough to avoid genuine changes in the measured variable. The primary criteria for choosing an appropriate interval are the stability of the response variable and the test's intended purpose. For example, if we were interested in the reproducibility of electromyographic measurements, it might be reasonable to test the patient on two occasions within one week. Range of motion measurements can often be repeated within one day or even within a single session. Measures of infant development might need to be taken over a short period, to avoid the natural changes that rapidly occur at early ages. If, however, we are interested in establishing the ability of an IQ test to provide a stable assessment of intelligence over time, it might be more meaningful to test a child using intervals of one year. The researcher must be able to justify the stability of the response variable to interpret test-retest comparisons.

Carryover and Testing Effects

With two or more measures, reliability can be influenced by the effect of the first test on the outcome of the second test. For example, practice or carryover effects can occur with repeated measurements, changing performance on subsequent trials. A test of dexterity may improve because of motor learning. Strength measurements can improve following warm-up trials. Sometimes subjects are given a series of pretest trials to neutralize this effect, and data are collected only after performance has stabilized. A retest score can also be influenced by a subject's effort to improve on the first score. This is especially relevant for variables such as strength, where motivation plays an important role. Researchers may not let subjects know their first score to control for this effect.

It is also possible for the characteristic being measured to be changed by the first test. A strength test might cause pain in the involved joint and alter responses on the second trial. Range of motion testing can stretch soft tissue structures around a joint, increasing the arc of motion on subsequent testing. When the test itself is responsible for observed changes in a measured variable, the change is considered a testing effect. Oftentimes, such effects will be manifested as systematic error, creating consistent changes across all subjects. Such an effect will not necessarily affect reliability coefficients, for reasons we have already discussed.

Reliability Coefficients for Test-Retest Reliability

Test-retest reliability has traditionally been analyzed using the Pearson product-moment coefficient of correlation (for interval-ratio data) or the Spearman rho (for ordinal data). As correlation coefficients, however, they are limited as estimates of reliability. The intraclass correlation coefficient (ICC) has become the preferred index, as it reflects both correlation and agreement. With nominal data, percent agreement can be determined and the kappa statistic applied. In situations where the stability of a response is questioned, the standard error of measurement (SEM) can be applied.

Rater Reliability

Many clinical measurements require that a human observer, or rater, be part of the measurement system. In some cases, the rater is the actual measuring instrument, such as in a manual muscle test or joint mobility assessment. In other situations, the rater must observe performance and apply operational criteria to subjective observations, as in a gait analysis or functional assessment. Sometimes a test necessitates the physical application of a tool, and the rater becomes part of the instrument, as in the use of a goniometer or taking of blood pressure. Raters may also be required simply to read or interpret the output from another instrument, such as an electromyogram, or force recordings on a dynamometer. However the measurements are taken, the individual performing the ratings must be consistent in the application of criteria for scoring responses.

This aspect of reliability is of major importance to the validity of any research study involving testers, whether one individual does all the testing or several testers are involved. Data cannot be interpreted with confidence unless those who collect, record and reduce the data are reliable. In many studies, raters undergo a period of training, so that techniques are standardized. This is especially important when measuring devices are new or unfamiliar, or when subjective observations are used. Even when raters are experienced, however, rater reliability should be documented as part of the research protocol.

To establish rater reliability the instrument and the response variable are considered stable, so that any differences between scores are attributed to rater error. In many situations, this may be a large assumption, and the researcher must understand the nature of the test variables and the instrumentation to establish that the rater is the true source of observed error.

Intrarater Reliability

Intrarater reliability refers to the stability of data recorded by one individual across two or more trials. When carryover or practice effects are not an issue, intrarater reliability is usually assessed using trials that follow each other with short intervals. Reliability is best established with multiple trials (more than two), although the number of trials needed is dependent on the expected variability in the response. In a test-retest situation, when a rater's skill is relevant to the accuracy of the test, intrarater reliability and test-retest reliability are essentially the same estimate. The effects of rater and the test cannot be separated out.

Researchers may assume that intrarater reliability is achieved simply by having one experienced individual perform all measurements; however, the objective nature of scientific inquiry demands that even under expert conditions, rater reliability should be evaluated. Expertise by clinical standards may not always match the level of precision needed for research documentation. By establishing statistical reliability, those who critique research cannot question the measurement accuracy of data, and research conclusions will be strengthened.

Rater Bias. We must also consider the possibility for bias when one rater takes two measurements. Raters can be influenced by their memory of the first score. This is most relevant in cases where human observers use subjective criteria to rate responses, but can operate in any situation where a tester must read a score from an instrument. The most effective way to control for this type of error is to blind the tester in some way, so that the first score remains unknown until after the second trial is completed; however, as most clinical measurements are observational, such a technique is often unreasonable. For instance, we could not blind a clinician to measures of balance, function, muscle testing or gait where the tester is an integral part of the measurement system. The major protections against tester bias are to develop grading criteria that are as objective as possible, to train the testers in the use of the instrument, and to document reliability across raters.

Interrater Reliability

Interrater reliability concerns variation between two or more raters who measure the same group of subjects. Even with detailed operational definitions and equal skill, different raters are not always in agreement about the quality or quantity of the variable being assessed. Intrarater reliability should be established for each individual rater before comparing raters to each other.

Interrater reliability is best assessed when all raters are able to measure a response during a single trial, where they can observe a subject simultaneously and independently. This eliminates true differences in scores as a source of measurement error when comparing raters' scores. Videotapes of patients performing activities have proved useful for allowing multiple raters to observe the exact same performance.^3,4,5 Simultaneous scoring is not possible, however, for many variables that require interaction of the tester and subject. For example, range of motion and manual muscle testing could not be tested simultaneously by two clinicians. With these types of measures, rater reliability may be affected if the true response changes from trial to trial. For instance, actual range of motion may change if the joint tissues are stretched from the first trial. Muscle force can decrease if the muscle is fatigued from the first trial.

Researchers will often decide to use one rater in a study, to avoid the necessity of establishing interrater reliability. Although this is useful for attempting consistency within the study, it does not strengthen the generalizability of the research outcomes. If interrater reliability of measurement has not been established, we cannot assume that other raters would have obtained similar results. This, in turn, limits the application of the findings to other people and situations. Interrater reliability allows the researcher to assume that the measurements obtained by one rater are likely to be representative of the subject's true score, and therefore, the results can be interpreted and applied with greater confidence.

Reliability Coefficients for Rater Reliability

The intraclass correlation coefficient (ICC) should be used to evaluate rater reliability. For interrater reliability, ICC model 2 or 3 can be used, depending on whether the raters are representative of other similar raters (model 2) or no generalization is intended (model 3). For intrarater reliability, model 3 should be used (see Chapter 26).

Alternate Forms

Many measuring instruments exist in two or more versions, called equivalent, parallel or alternate forms. Interchange of these alternate forms can be supported only by establishing their parallel reliability. Alternate forms reliability testing is often used as an alternative to test-retest reliability with paper-and-pencil tests, when the nature of the test is such that subjects are likely to recall their responses to test items. For example, we are all familiar with standardized tests such as the Scholastic Aptitude Test (SAT) and the Graduate Record Examination (GRE), professional licensing exams or intelligence tests, which are given several times a year, each time in a different form. These different versions of the tests are considered reliable alternatives based on their statistical equivalence. This type of reliability is established by administering two alternate forms of a test to the same group, usually in one sitting, and correlating paired observations. Because the tests are ostensibly different, they can be given at relatively the same time without fear of bias from one to the other. Although the idea of alternate forms has been applied mostly to educational and psychological testing, there are many examples in clinical practice. For example, clinicians use parallel forms of gait evaluations, tests of motor development, strength tests, functional evaluations, and range of motion tests. Many of these have not been tested for alternate forms reliability.

The importance of testing alternate forms reliability has been illustrated in studies of hand dynamometers. Several models are available, each with slightly different design features. Because these tools are often used to take serial measurements, patients might appear to be stronger or weaker simply because of error if different instruments were used. Studies comparing various models have shown that some instruments generate significantly different strength scores,⁶ while others have shown comparable values.⁷ Establishing this method comparison is necessary if absolute values are to be compared or equated across tests, and to generalize findings from one study to another or from research to practice.

Reliability Coefficients for Alternate Forms Reliability

Correlation coefficients have been used most often to examine alternative forms reliability. The determination of limits of agreement has been proposed as a useful estimate of the range of error expected when using two different versions of an instrument. This estimate is based on the standard deviation of difference scores between the two instruments (see Chapter 26).

Internal Consistency

Software instruments, such as questionnaires, written examinations and interviews are ideally composed of a set of questions or items designed to measure particular knowledge or attributes. Internal consistency, or homogeneity, reflects the extent to which items measure various aspects of the same characteristic and nothing else. For example, if a professor gives an exam to assess students' knowledge of research design, the items should reflect a summary of that knowledge; the test should not include items on anthropology or health policy. If we assess a patient's ability to perform daily tasks using a physical function scale, then the items on the scale should relate to aspects of physical function only. If some items evaluated psychological or social characteristics, then the items would not be considered homogeneous. The scale should, therefore, be grounded in theory that defines the dimension of physical function, thereby distinguishing it from other dimensions of function.

The most common approach to testing internal consistency involves looking at the correlation among all items in a scale. For most instruments, it is desirable to see some relationship among items, to reflect measurement of the same attribute, especially if the scale score is summed. Therefore, for inventories that are intended to be multidimensional, researchers generally establish subscales that are homogenous on a particular trait (even though items are often mixed when the test is administered). For example, the Short-Form 36-item (SF-36) health status measure is composed of eight subscales, including physical function, limitations in physical role, pain, social function, mental health, limitations in emotional role, vitality and general health perception.⁸ Each of these subscales has been evaluated separately for internal consistency.⁹

Split-Half Reliability

If we wanted to establish the reliability of a questionnaire, it would be necessary to administer the instrument on two separate occasions, essentially a test-retest situation. Oftentimes, the interval between testing is relatively brief, to avoid the possibility for true change. Recall of responses, then, becomes a potential threat, as it might influence the second score, making it impossible to get a true assessment of reliability. One solution to this problem is the use of parallel forms, but this shifts the measure of reliability to a comparison of instruments, rather than reliability of a single instrument.

A simpler approach combines the two sets of items into one longer instrument, with half the items being redundant of the other half. One group of subjects takes the test at a single session. The items are then divided into two comparable halves for scoring, creating two separate scores for each subject. Typically, questions are divided according to odd and even items. This is considered preferable to comparing the first half of the test with the second half, as motivation, fatigue and other psychological elements can influence performance over time, especially with a long test. Reliability is then assessed by correlating results of two halves of the test. If each subject's half-test scores are highly correlated, the whole test is considered reliable. This is called split-half reliability. This value will generally be an underestimate of the true reliability of the scale, since the reliability is proportional to the total number of items in the scale. Therefore, because the subscales are each half the length of the full test, the reliability coefficient is too low.

The obvious problem with the split-half approach is the need to determine that the two halves of the test are actually measuring the same thing. In essence, the two halves can be considered alternate forms of the same test; however, the split-half method is considered superior to test-retest and alternate forms procedures because there is no time lag between tests, and the same physical, mental and environmental influences will affect the subjects as they take both sections of the test.

Reliability Coefficients for internal Consistency

The statistic most often used for internal consistency is Cronbach's coefficient alpha (α).¹⁰ This statistic can be used with items that are dichotomous or that have multiple choices.^* Conceptually, coefficient α is the average of all possible split-half reliabilities for the scale. This statistic evaluates the items in a scale to determine if they are measuring the same construct or if they are redundant, suggesting which items could be discarded to improve the homogeneity of the scale. Cronbach's α will be affected by the number of items in a scale. The longer the scale, the more homogeneous it will appear, simply because there are more items.

For split-half reliability, the Spearman-Brown prophecy statistic is used as an estimate of the correlation of the two halves of the test.

We can also assess internal consistency by conducting an item-to-total correlation; that is, we can examine how each item on the test relates to the instrument as a whole. To perform an item-to-total correlation, each individual item is correlated with the total score, omitting that item from the total. If an instrument is homogeneous, we would expect these correlations to be high. With this approach it is not necessary to create a doubly long test. The Pearson product-moment correlation coefficient is appropriate for this analysis (see Chapter 23).

Measurement in clinical research or practice is never used as an end unto itself. Measurements are used as information for decision making, evaluation or prediction. The score we obtain from a test is given a meaning beyond the specific situation in which it was taken; that is, we make generalizations about performance or behavior based on measurements. In many ways, reliability provides the foundation for making such generalizations, as we must have confidence in the dependability of measurements if they are to be applied in different situations or used to make decisions for future action.

On this basis, Cronbach and his colleagues¹¹ introduced the idea that reliability theory should be more accurately conceptualized in terms of generalizability theory. They suggested that every individual score can be thought of as a sample from a universe of possible scores that might have been obtained under the same testing conditions. These specific testing conditions define the universe to which measures of reliability can be generalized. For example, we can test the grip strength of a patient with rheumatoid arthritis using a hand dynamometer. A single measured score of 15 pounds would be one of a universe of possible scores that might have been obtained under the same testing conditions, using the same dynamometer, at the same time of day, and by the same examiner. Because most research involves taking small samples of measurements, we assume that our observations are representative of this infinite distribution of possible scores. A single measurement then becomes the best estimate of a true score under those testing conditions. Reliability is essentially a measure of how good an estimate that measurement is.

According to classical reliability theory, an individual's observed score can be partitioned into a true component and an error component. The true score is assumed to be a fixed value that exists independently of any other conditions of measurement. Therefore, any differences between the observed score and the true score are due to random error (see Figure 5.3A). This theory also assumes that the error component is undifferentiated; that is, it comes from many different sources in an unbiased form. If we accept this premise, then we should be comfortable applying a given reliability estimate to any other situation where the same measurement is taken, because the error in each situation should be equally random.

In generalizability theory, however, the conditions of testing are not considered independent factors; that is, the true score is a function of an underlying theoretical component only as it exists under specific conditions. This means that not all variations from trial to trial should be attributed solely to random error. If we can identify relevant testing conditions that influence test scores, then we should be able to explain and predict more of the variance in a set of scores, effectively leaving less variance unexplained as error.

Facets of Reliability

The concept of generalizability, therefore, forces us to interpret reliability within a multidimensional context, that is, in relation to a set of specific testing conditions. Each condition that defines this context is called a facet. A particular combination of facets characterizes the universe to which reliability can be generalized. The researcher determines which facets are relevant to the measurement of a particular variable. By specifying those facets of greatest relevance, it is possible to statistically determine how much of the variance in observed scores can be attributed to each facet (see Figure 5.3B).

For instance, suppose we set up a test-retest situation for goniometric measurements of elbow flexion. In one scenario, we obtain two sets of ratings taken by one rater on a sample of 10 patients. By comparing these two sets of scores, we can establish the reliability of data across two occasions. Therefore, "occasion" becomes a facet of reliability. These occasions may be trials within one session or over two separate days. How these occasions are defined determines how the results can be generalized. Similarly, if we collect data from four clinicians' ratings within a single session, we can establish the reliability of data across four raters. Therefore, "rater" becomes a facet of reliability. In this case, we would have to establish the criteria that would characterize these raters, such as their years of experience or special training. Then results can be generalized to other raters with similar training.

The concept of generalizability is a clinically useful one, especially in terms of the development and standardization of measurement tools, because it provides a frame of reference for interpretation of reliability coefficients and suggests areas of improvement of protocols. Because we cannot expect to verify the reliability of every instrument on every patient for every rater, we must be able to document those characteristics that are relevant to generalizing measurement consistency. Generalizability theory also emphasizes that reliability is not an inherent quality of an instrument, but exists only within the context in which it was tested. Accordingly, we cannot automatically assume that estimates of reliability from one study can be applied to other raters, environments, testing conditions, or types of patients, unless we specifically address these factors in our analysis. Most reliability studies look at rater as the most important single facet.

The essence of generalizability theory suggests, however, that other facets must be examined before an instrument's reliability can be fully understood.

In effect, then, there is no one coefficient that provides a complete estimate of reliability for a given test or measurement. Separate coefficients that address different facets can be obtained and applied to relevant situations. Without such documentation, it is not possible to make reasonable claims of general reliability for any instrument. Statistical and design considerations that are appropriate for generalizability studies are discussed further in Chapter 26.

Minimal Detectable Difference

Measurement error is of special interest in the assessment of change. When we take two measurements of the same variable at different times, we are usually doing so to determine if there is a difference in value, perhaps due to a specific intervention or the passage of time. When we observe a difference in the measurement, we want to assume that it represents a change in the true score—the variable really did change. But because all measurements are subject to some degree of error, the accuracy of this assumption will depend on the reliability of the measurement.

We can appreciate, then, that when we measure a difference between two scores, some portion of that change may be error, and some portion may be real. The concept of a minimal detectable difference (MDD) has been used to define that amount of change in a variable that must be achieved to reflect a true differenced.^† This is the smallest amount of difference that passes the threshold of error for a specific instrument and application.¹² It is the smallest amount of change an instrument can accurately measured.^‡ The greater the reliability of the measurement, the smaller the MDD. The precision and level of measurement will also affect this detectable difference.

Methodological studies that focus on an instrument's ability to measure change will report the MDD as an important characteristic of the measurement tool. By knowing the MDD, clinicians can generalize the information from such a report to their specific clinical situation. To illustrate this concept, van der Esch et al.¹³ studied measurement of knee joint laxity in healthy subjects using an instrumented system. Based on test-retest measures, they determined that the MDD was 3.73 degrees, with a 95% confidence interval of 2.66–6.37. This means that we can be 95% confident that measurements outside of this range would represent true change. They also found that the mean score for knee joint laxity was approximately 6 degrees. Therefore, the researchers concluded that it would be difficult to interpret measurement of change since the observed change and the expected error were in the same range. It would not be possible to distinguish error from true joint change using their instrument. Statistical application of the MDD is discussed in Chapters 26 and 27.

Population-Specific Reliability

The concept of generalizability also emphasizes the need to consider characteristics of those involved in establishing reliability. This includes the patients or subjects being tested and the raters who do the testing. Reliability that is established on subjects from one population cannot automatically be attributed to other populations. This has been termed population-specific reliability. Clearly, factors such as pain, deformity, weakness, anxiety, and spasticity can alter the way a patient responds to a measurement and the consistency with which a clinician can take those measurements. Doing a manual muscle test on a patient with shoulder pain may be a different experience from doing the test on someone with limited range of motion. Range of motion measurements may be difficult to standardize among patients with joint deformities, joint pain or tenderness, or severe limitations of movement. Similarly, rater reliability must take into account the skill, experience and training of the individual performing the test. Therefore, reliability of measurement must be documented according to the characteristics of a specific group of individuals who will be part of the measurement.

When choosing an instrument for clinical or research measurement, it would certainly make sense to use a tool for which reliability has already been demonstrated. Even then, however, there is no guarantee that the same degree of reliability will be achieved in every situation. Therefore, researchers often perform pilot studies to establish reliability prior to the start of actual data collection. This should be a routine part of the research process, especially when observational measurements are used. It allows the researcher to determine if raters are adequately trained to obtain valid measurements. It may be important to consider, however, that specific characteristics of the pilot test situation may be quite different from those that will be encountered during data collection. At the least, raters are often much more careful when they know the scores will be examined for reliability. Mitchell suggests that actual data collection should include multiple trials of each measurement, and that these trials should be assessed for reliability as part of the study's data analysis.¹⁴ Although training and testing prior to data collection are essential for developing measurement accuracy, the validity of an experiment will be well served by documenting the error rate of measurement within the actual data used for analysis. For generalization of findings, it is probably more meaningful to report the reliability of these data than those obtained during pilot testing.