Chapter 15: Surveys and Questionnaires

One of the most popular methods for collecting descriptive or subjective data is the survey approach. A survey is composed of a series of questions that are posed to a group of subjects, and may be conducted as an oral interview or as a written or electronic questionnaire. Sometimes the data are intended for generalization to a larger population; other times they may be intended as a description of a particular group. Surveys in clinical research are often concerned with describing current practices, attitudes and values, or characteristics of specific groups. For example, survey questionnaires have been used to compare the effectiveness of medication, acupuncture and spinal manipulation for chronic low back pain,¹ to study physicians' attitudes and practices toward disclosure of prognosis for terminally ill patients,² and to describe the demographics and injury characteristics of patients with spinal cord injury.³ Standardized questionnaires are also used extensively as instruments for assessing outcomes related to function, health status and quality of life. As these examples illustrate, survey data can be used in experimental, exploratory or descriptive studies.

The purpose of this chapter is to present an overview of the structure of survey instruments. We discuss essential elements of survey design, question writing, and some special assessment techniques associated with questionnaires, including several measurement scales.

Interviews

In an interview the researcher asks respondents specific questions and records their answers for later analysis. Interviews can take a few minutes or several hours, depending on the nature of the questions and the respondent's willingness to share information. Interviews can be conducted face to face or over the telephone, although face-to-face interviews tend to be more effective for establishing rapport between the interviewer and the respondent. This interaction can be important for eliciting forthright responses to questions that are of a personal nature. The advantage of the interview approach is the opportunity for in-depth analysis of respondents' behaviors and opinions because the researcher can probe responses and directly observe respondents' reactions. The major disadvantages of interviews include cost and time, the need for personnel to carry out the interviews, scheduling and the lack of anonymity of the respondents.

Most interviews are structured in that they consist of a standardized set of questions that will be asked. In this way, all respondents are exposed to the same questions, in the same order, and are given the same choices for responses. In an unstructured interview, the interviewer does not have a fixed agenda, and can proceed informally to question and discuss issues of concern. This format is typically conversational and is often carried out in the respondent's natural setting. Many qualitative studies use the unstructured interview approach to generate data.

Questionnaires

Questionnaires are structured surveys that are self-administered using pen and paper or electronic formats. The advantages of using questionnaires are many. They are generally more efficient than interviews because respondents complete them on their own time. Data can be gathered from a large sample in a wide geographical distribution in a relatively short period of time. Written forms are standardized, so that everyone is exposed to the same questions in the same way, reducing potential bias from interactions with an interviewer. Respondents to questionnaires can take time to think about their answers and to consult records for specific information. Questionnaires also provide anonymity, encouraging honest and candid responses. Questionnaires are particularly useful as a research method for examining phenomena that can be assessed through self-observation, such as attitudes and values. They are not as useful for studying behaviors that require objective observation. The primary disadvantages of the written questionnaire are the potential for misunderstanding or misinterpreting questions or response choices, and unknown accuracy or motivation of the respondent. In interviews, the researcher can clarify such misinterpretations.

The most common method of distributing questionnaires has traditionally been through the mail, although many research situations allow for in-person distribution. Electronic distribution of surveys is quickly becoming common practice. Survey software is available through many vendors, allowing for anonymity and automatic tallying of responses. Such questionnaires are economical and can reach a large population in a relatively short period.

A major disadvantage of mail or email questionnaires is that the return rate is often quite low. Responses from 60% to 80% of a sample are usually considered excellent. Realistically, researchers can expect return rates between 30% and 60% for most studies. Actual response rates are lowered further by having to discard returns that are incomplete or incorrectly filled out. Low returns can severely limit the external validity of survey results. Therefore, survey samples are usually quite large so that a sufficient percentage of usable responses will be obtained.

Self-Report

Survey data that are collected using either an oral interview or a written questionnaire are based on a form of self-report; that is, the researcher does not directly observe the respondent's behavior or attitudes, but only records the respondent's report of them. There is always some potential for bias or inaccuracy in self-reports, particularly if the questions concern personal or controversial issues. The phenomenon of recall bias can be a problem when respondents are asked to remember past events, especially if these events were of a sensitive nature. Research has shown, however, that self-report measures are generally valid. For instance, variables such as injury,⁴ mobility function,⁵ hypertension,⁶ and smoking habits^7,8 have been reported accurately, although a few studies have shown a poor correlation between performance and self-report measures.^9,10 These differences point out the need to understand the target population and the respondents' abilities to answer the questions posed. For many variables, however, such as perceptions, fears, motivations and attitudes, self-report is the only direct way to obtain information.

The process of developing a survey instrument is perhaps more time consuming than most people realize. It involves several stages, within which the instrument is written and revised, until it is finally ready for use as a research tool.

The Research Question

The first consideration in every research effort is delineation of the overall research question (see Chapter 7). The research problem must be identified with reference to a target population. These decisions will form the structure for deciding the appropriate research design. A survey is appropriate when the question requires obtaining information from subjects, rather than measuring performance.

Guiding Questions

As with any other research approach, validity is a major concern in the design of a survey instrument; that is, the questionnaire or interview must measure what it was intended to measure. Questions are not asked out of casual interest or curiosity, but because they reflect specific pieces of information that taken as a whole will address the proposed research question. Therefore, the first step in developing the survey is to define its purpose through a series of guiding questions, or objectives that delineate what the researcher is trying to find out. The guiding questions may reflect purely descriptive interest, or they may address expected relationships among variables. For example, Couch and colleagues¹¹ examined the role that play occupies within occupational therapy practice with preschoolers. They proposed three guiding questions for their study:

These types of questions focus the content of a questionnaire.

Hypotheses

Survey instruments can also be designed to examine relationships. Some researchers will, therefore, specify hypotheses in addition to guiding questions. Hypotheses are important to direct statistical analyses and conclusions. To illustrate, Rozier and coworkers¹² looked at the relationship between gender and career success factors in physical therapists. One of the several guiding questions they posed was, "Are perceptions of career success different for male versus female physical therapists?" They proposed a series of hypotheses to answer this question, such as:

By delineating the specific variables and expected outcomes, the authors were able to clearly structure the analysis and discussion of their findings.

Questionnaire Outline

Once the guiding questions or hypotheses are formulated, the researcher must develop a detailed outline, listing each item of information that will be needed to answer the guiding questions. Each item should relate back to at least one of the study's objectives. Often, more than one item will be needed to address a single question or hypothesis.

Stating useful guiding questions that support an overall research question requires that the researcher have a clear conceptualization of the phenomenon or characteristics being studied. In the study by Couch,¹¹ it would not be possible to ask questions that allow the researchers to determine if play was truly part of a therapist's practice without a good concept of "play." In Rozier's study,¹² the elements of career success must be understood to compare opinions of male and female respondents. Each individual question in a survey should add to a larger informational context that will answer the guiding questions.

Typically, researchers also include questions about important demographic information in a survey. Couch et al¹¹ asked occupational therapists about their academic degrees, years of experience and practice setting. Rozier et al¹² asked their sample for information about age, employment setting, length of employment and family responsibilities. Studies will also often include items related to income, race, marital status, living situation and so on. This type of information is needed to describe the characteristics of the respondents, to compare the characteristics of the sample with those of the population to which the results will be generalized, and to interpret how personal characteristics are related to the subject's responses. Guiding questions should be included to reflect how this information will be related to the overall research question.

Review of Existing Instruments

The next step in questionnaire development should be to review existing instruments, to determine if they are applicable or adaptable for the study. Many investigators have developed and validated instruments for a variety of purposes. For example, instruments have been developed for exploring attitudes,¹³ health behaviors,¹⁴ and functional status.^15,16 Often, these instruments can be borrowed directly or in part, or they can be modified to fit new research situations, saving a great deal of time. It is not wise to adopt previously used surveys blindly, without considering differences in populations being studied and the specific objectives of the instrument. It is, however, always possible to benefit from the review of literature and insights into development or validation issues that the creator of such an instrument can provide.

Designing the Instrument

The researcher begins to design a survey by writing a series of questions that address each behavior, knowledge, skill or attitude reflected in the guiding questions. Questions should be grouped and organized to reflect each category or topic. The first draft of a questionnaire should include several questions for each topic, so that these can eventually be compared and weeded out. Content should flow so that the respondent's thought processes will follow a logical sequence. Questions should proceed from the general to the specific. The format of questions will vary, depending on how the survey will be administered, that is, by phone or personal interview, or questionnaire. The initial questions should pique the respondent's interest, or at least be "neutral." Sensitive questions should come later. Some researchers put demographic questions at the beginning, but many prefer to keep these less interesting questions for the end.

The organization of the survey is extremely important to the success of its application. Respondents can easily be turned off by a format that is complicated or confusing. The document should be presented in as "friendly" a format as possible. The page should be uncluttered, printed in laser-quality print, and aligned so that it is easy to find the next question. The font size should be at least 11 or 12 point. The font should be simple to read, not fancy or unusual. Some researchers like to use colored paper, rather than white, for written questionnaires to make the survey stand out. Colors used in email surveys should be subtle.

Preliminary Drafts

The preliminary draft of the survey should now be distributed to a panel of colleagues who can review the document, identify problems with questions, including wording and organization. Ask for criticism and suggestions for constructive change. No matter how carefully the survey has been designed, the researcher is usually too close to it to see its flaws. Provide the panel with the study's guiding questions. Based on the panel's comments, the survey should be revised, and then presented to the panel again for further comment. The reviewers should try to answer the survey questions, and discuss how they interpreted each one. The revision process should continue, with additional feedback from evaluators, until the researcher is satisfied that the instrument is concise, clear and serves its intended purpose. This process is indeed time consuming, but necessary, and helps to establish the content validity of the instrument.

Pilot Testing and Revisions

The revised questionnaire should then be pilot tested on a small representative sample, perhaps 5 to 10 individuals from the target population. The researcher should interview these respondents to determine where questions were unclear or misleading. If the researcher is unsure about the appropriateness of specific wording, several versions of a question can be asked to elicit the same information in different ways, and responses can be compared for their reliability. Look for missing answers and inconsistencies. It is also useful to monitor the time it takes for respondents to complete the questionnaire. It may be helpful to administer the survey to this group on two occasions, perhaps separated by several days, to see if the responses are consistent, as a way of estimating test- retest reliability. Based on the results of pilot testing, the questionnaire may again be revised and retested until the final instrument attains an acceptable level of validity.

A major concern during this process will be the length of the survey. More often than not, the initial versions will be too long. Long questionnaires are less likely to maintain the respondent's attention and motivation, resulting in potentially invalid or unreliable responses or, in the case of mail surveys, nonresponses. The importance of each item for the interpretation of the study should be examined, and only those questions that make direct and meaningful contributions should be retained. Researchers have shown that shorter questionnaires are often more valid than longer ones, as items generally have some redundancy built into them.¹⁷

If the study involves the use of interviewers, a formal training process should be incorporated once the questions have been finalized. Interviewers must be consistent in how they present the survey, how questions are asked, and how probing follow-up questions are used (if they are to be allowed). They should be briefed on the purpose of the study, and their presentation should convey the proper attitude. The interviewers must understand the process of recording responses, an important skill when open- ended questions are used.

Selecting a Sample

Before the survey can be administered, the researcher must choose a sample. An accessible population must be identified. As much as possible, a probability sample should be selected. Stratified sampling is often used to control for variations within the sample, such as geographical area when national samples are used. Cluster sampling may be used to increase accessibility of respondents.

For interview surveys, the respondents will typically be within a local geographic area, and may be recruited from agencies or clinics. Before an interview is administered, the potential respondents should be contacted to elicit their cooperation. For telephone interviews, it is appropriate to send advance notice in the mail that the phone call will be coming as a means of introduction and as a way of establishing the legitimacy of the phone call. For mail surveys, the accessible population may be quite dispersed, and may be limited only by the availability of mailing addresses. Mailings lists can be purchased from professional associations or organizations. Published lists of schools or hospitals can usually be obtained from libraries or professional organizations.

Contacting Respondents

Survey respondents should be given an introduction to the survey so that they understand its purpose and how the data will be used. They should be given an idea of how long it will take to complete the survey. In a written questionnaire, this information will be included in the cover letter. In a mail survey, a self-addressed stamped envelope must be included. It is also appropriate to ask respondents if they would like a copy of the study results when it is completed. Because the survey will typically be anonymous, a separate form can be included for them to send back with their name and address for this purpose.

Because of low response rates in surveys, the researcher should plan to follow up on those who do not respond. To maintain anonymity, in mail surveys, researchers will often code the back of the return envelope, so they can keep a record of who has returned the questionnaire. It is appropriate to send out reminders—postcards, emails, or phone calls—about 2 weeks after the initial mailing to encourage a reply. Because online survey responses are anonymous, email reminders can be sent to the entire sample, (with apologies to those who did respond). Although the majority of responses will be obtained within the first 2 weeks, a reasonable improvement can usually be obtained through follow-up.

Cover Letter

Questionnaires must include a cover letter that orients the respondents to the survey and politely requests their participation. Because a questionnaire can easily be ignored, the cover letter becomes vitally important to encourage a return. An example of a cover letter is shown in Figure 15.1. The letter should include the following elements:

1. Start with the purpose of the study, including its importance. If the research is sponsored by an agency, this information should be included. If the project is a thesis or student project, the respondents should know this.

2. Indicate why the respondent has been chosen for the survey.

3. Assure the respondents that the survey will be anonymous. Encourage them to be honest in their answers, and assure them that they can refuse to answer any questions that make them uncomfortable.

4. Suggest how long it will take to complete the questionnaire.

5. Ask them to respond by sending back the survey in the enclosed self-address stamped envelope or by electronic submission. Provide a deadline date. It is reasonable to give 2 to 3 weeks for a response. A shorter time is an imposition, and longer may result in the questionnaire being put aside and forgotten.

6. Thank respondents for their cooperation. Stress the importance of their response for your work. Provide an opportunity for them to receive a summary of the report.

7. Sign the letter (or use an electronic signature), including your name, degrees and affiliation. If there are several investigators, it is appropriate to include all signatures.

Two types of questions can be asked in a survey: open-ended and closed-ended questions. Open-ended questions ask respondents to answer in their own words. Closed-ended questions provide multiple response choices.

Open-Ended Questions

Open-ended questions are useful for probing respondents' feelings and opinions, without biases or limits imposed by the researcher. For example, "What aspects of your job are most satisfying to you?" would require the respondent to provide specific examples of job characteristics. This format is useful when the researcher is not sure of all possible responses to a question. Therefore, respondents are given the opportunity to provide answers in their own words and from their own perspective. Sometimes researchers will use open-ended questions in a pilot study to determine a range of responses which can then be converted to a multiple choice item.

Open-ended questions are, however, difficult to code and analyze because so many different responses can be obtained. If open-ended questions are misunderstood, they may elicit answers that are essentially irrelevant to the researcher's goal. Respondents may not want to take the time to write a full answer, or they may answer in a way that is clear to them but uninterpretable, vague, or incomplete to the researcher. For instance, a question like, "What types of exercise do you do regularly?" could elicit a long list of specific movements or a general description of an exercise routine. If the purpose of the question is to find out if the respondent engages in particular exercise activities, it would be better to list certain exercises and ask if they are done. Open-ended questions can be effective in interviews because the interviewer can clarify the respondent's answers by follow-up questions. Open-ended questions are generally avoided in questionnaires, except where responses are fairly objective, such as asking for a person's yearly income, or where the researcher's purpose is to explore respondents' motivations or behaviors without presenting a predefined list of choices.

Closed-Ended Questions

Closed-ended questions ask respondents to select an answer from among several choices that are provided by the researcher.

This type of question is easily coded and provides greater uniformity across responses. Its disadvantage is that it does not allow respondents to express their own personal viewpoints and, therefore, may provide a biased response set. The list of choices may overlook some important responses, or they may bias answers by presenting a particular attitude.

There are two basic considerations in constructing closed-ended questions. First, the responses should be exhaustive; that is, they should include all possible responses that can be expected. As a protection, it is often advisable to include a category for "not applicable" (NA), "don't know," or "other (please specify___)." In the preceding example, for instance, it is likely that respondents will have other reasons for job satisfaction that have not been listed. Second, the response categories should be mutually exclusive; that is, each choice should clearly represent a unique answer. The preceding example is inadequate on this criterion. For example, a respondent may see "intellectual challenge" and "opportunities for educational growth" as elements of the same concept. Where only one response is desired, it may be useful to add an instruction asking respondents to select the one best answer or the answer that is most important; however, this technique does not substitute for carefully worded questions and choices.

There should also be a rationale for ordering response choices. Sometimes, there is an inherent hierarchy in the responses, so that choices can be given in order of increasing or decreasing intensity or agreement. When there is no purposeful order, the researcher must be careful to avoid "leading" the respondent to a particular choice by the order or phrasing.

Sometimes the researcher is interested in more than one answer to a question. For instance, we might want to know all the reasons for a person's job satisfaction. Instructing the respondent to mark "all that apply" creates an interpretation problem in that the respondent may not choose a particular item because it does not apply, because it was not clear, or because it was missed. In addition, it is difficult to code multiple responses. When multiple choices are of interest, it is better to ask respondents to mark each choice separately, as in the following example:

Format of Closed-Ended Questions

Two common formats are used to list choices to closed-ended questions: (1) using brackets, as just presented, where respondents are asked to check the appropriate response, or (2) asking respondents to circle the number or letter that appears before the answer, as shown next.

The simplest form of closed-ended question is one that presents two choices, or dichotomous responses:

When questions address a characteristic that is on a continuum, such as attitudes or quality of performance, it is more useful to provide a range of responses, so that the respondent can find a choice that represents the appropriate intensity of response. Usually, three to five multiple-choice options are provided. An option for "Don't know" or "Unsure" should always be included.

When a series of questions use the same format, a grid or checklist can provide a more efficient presentation. With this approach, instructions for using the response choices need only be given once, and the respondent can quickly go through many questions without reading a new set of choices. For example, Figure 15.2 shows a checklist for a question concerning patients' level of knee pain during different activities.

An alternative question format is the rank-order question, where the respondent is presented with a series of responses and is asked to rank the responses on an ordinal scale.

Some question sequences try to follow up on specific answers with more detailed questions, using a technique called branching. Depending on the response to an initial question, the respondent will be directed to answer additional questions or to skip ahead to a later question.

This process saves time by avoiding questions that are irrelevant to a specific respondent.

Wording Questions

Simplicity is a key to good questionnaires. Sentences should be succinct and grammatically correct. Respondents will not be inclined to ponder the meaning of a long, involved question. The author of the questionnaire should assume that respondents will read and answer questions quickly and should provide choices that will be understood at a glance. Questions should be written in common language for the lowest educational level that might be encountered.

Language may be an issue if respondents do not speak English, or English is a second language. The researcher must know the sample well enough to accommodate language in the wording of questions. Idioms or subtle cultural expressions should be carefully avoided. The researcher may have the questionnaire translated into another language for specific sample groups. The translation must account for cultural biases.

It goes without saying that survey questions must be clear and unambiguous. Questions that require subtle distinctions to interpret responses are more likely to be misunderstood. For example, consider the question: "How many different sports do you participate in?" There are two ambiguous terms here. First, there may be different ways to define sports. Some may include any form of physical activity, including riding a stationary bicycle; others may include only legitimate field sports. Second, what constitutes participation? Does it have to be an organized schedule of play, or can it mean throwing a basketball in your yard on weekends or playing golf once a year? This type of ambiguity can be corrected by providing the respondent with appropriate definitions, as in the following example:

Double-Barreled Questions

Each question should be confined to a single idea. Surveys should avoid the use of double-barreled questions, using "or" or "and" to assess two things within a single question. For instance, "How many times a week do you jog or ride a stationary bicycle?" It is obviously possible to perform both of these activities at different rates, making it impossible to answer the question. It is better to ask two questions to assess each activity separately.

Frequency and Time Measures

Researchers are often interested in quantifying behavior in terms of frequency and time. For example, it might be of interest to ask, "How many alcoholic drinks do you consume each day?" or "How many patients do you treat per day?" These types of questions may be very difficult to answer because the frequency of the behavior may vary greatly from day to day, month to month, or even season to season. The researcher should determine exactly what aspect of the behavior is most relevant to the study and provide an appropriate time frame for interpreting the question. For instance, the question could ask about a particular period, such as the maximum number of patients seen within the last week, or the respondent can be asked to calculate an average daily value. This assumes, of course, that this time period is adequately representative for purposes of the study. Alternatively, the question could ask for an estimate of "typical" or "usual" behaviors. This approach makes an assumption about the respondent's ability to form such an estimate. Some behaviors are much more erratic than others. For example, it may be relatively easy to estimate the number of patients treated in a day, but it may be more difficult to estimate typical behavior in consuming alcoholic beverages. Estimates of "typical" will also tend to ignore extremes, which may or may not be important to the purpose of the study.

Questions related to time should also be specific. A question such as, "Has your back pain limited your ability to work?" may be difficult to answer if it is not a consistent problem. It is better to provide a time frame for reference. For example, "Has your back pain limited your ability to work within the past month?" Many function, pain and health status questionnaires specify time periods within the last month, last week, or last 24 hours.

Dealing with Sensitive Questions

Questionnaires often deal with sensitive or personal issues that can cause some discomfort on the part of the respondent. Although some people are only too willing to express personal views, others are hesitant, even when they know their responses are anonymous. Some questions may address social behaviors that have negative associations, such as smoking, sexual practices and drinking alcohol; others may inquire about behaviors that respondents are not anxious to admit to, such as ignorance of facts they feel they should know and compliance with medications or exercise programs. Sensitive questions may also be subject to recall bias. For example, respondents may be selective in their memory of risk factors for disease or disability. Respondents should be reminded in the introduction to the survey that they may refuse to answer any questions.

Sensitive questions should be phrased to put the respondent at ease. It may be useful to preface such questions with a statement as in these examples: "Many people forget to take their medications from time to time." "It is often very difficult for people to fit exercise sessions into their daily routines." "Clinicians are faced with a tremendous task in keeping up with the variety of treatment approaches that are being developed for low back pain. Depending on experience and practice, some clinicians have had an opportunity to learn these techniques more than others." These statements tell the respondent that it is okay if they fit into that category.

Phillips¹⁸ suggests that questions that ask respondents to admit to socially unacceptable behaviors should be phrased in a manner that assumes the respondent engages in the behavior. For instance, rather than asking,

we could ask one question:

A scale is an ordered system based on a series of questions or items that provide an overall rating that represents the degree to which a respondent possesses a particular attitude, value or characteristic. The purpose of a scale is to distinguish among people who demonstrate different intensities of the characteristic that is being measured. Scales have been developed to measure attitudes, function, health and quality of life, pain, exertion and other physical, physiological and psychological variables.

Categorical scales are based on nominal measurement. A question asks the respondent to assign himself according to one of several classifications. This type of scale is used with variables such as gender, diagnosis, religion or race. These data are expressed as frequency counts or percentages.

Most scales represent a characteristic that exists on a continuum. Continuous scales may be measured using interval or ratio values, such as age, blood pressure or years of experience. An ordinal scale requires that a continuous variable be collapsed into ranks. For instance, pain can be measured as "minimal, moderate, severe," or function as "independent, minimal assist, moderate assist, maximal assist, dependent." Scale items should represent the full range of values that represent the characteristic being measured.

Scales are created so that a summary score can be obtained from a series of items, indicating the extent to which an individual possesses the characteristic of interest. Because item scores are combined to make this total, it is important that the scale is structured around only one dimension; that is, all items should reflect different elements of a single characteristic. A summative scale is one that presents a total score with all items contributing equal weight to the total. A cumulative scale demonstrates an accumulated characteristic, with each item representing an increasing amount of the attribute being measured.

We will describe several scaling models used to summarize respondent characteristics: Likert scales, the semantic differential, visual analogue scales, cumulative scales and Rasch models.

Likert Scales

A Likert scale is a summative scale, most often used to assess attitudes or values. A series of statements is presented expressing a viewpoint, and respondents are asked to select an appropriately ranked response that reflects their agreement or disagreement with each one. For example, Figure 15.3 shows a set of statements that evaluate students' opinions about including a research course in an entry-level professional curriculum. Likert's original scale included five categories: strongly agree (SA), agree (A), neutral (N), disagree (D), and strongly disagree (SD).¹⁹ Many modifications to this model have been used, sometimes extending it to seven categories (including "somewhat disagree" and "somewhat agree") or four categories (eliminating "neutral").

There is no consensus regarding the number of response categories that should be used. Some researchers believe the "neutral" option should be omitted so that the respondents are forced to make a choice, rather than allowing them an "out" so that they do not have to take sides on an issue. Others feel that respondents who do not have strong feelings should be given a viable option to express that attitude. When the forced choice method is used, responses that are left blank are generally interpreted as "neutral."

Each choice along the scale is assigned a point value, based on the degree to which the item represents a favorable or unfavorable characteristic. For example, we could rate SA = 5, A = 4, N = 3, D = 2, SD = 1, or we could use codes such as SA = 2, A = 1, N = 0, D = −1, SD = −2. The actual values are unimportant, as long as the items are consistently scored; that is, agreement with favorable items should always be scored higher than agreement with unfavorable items. Therefore, if positively phrased items are coded 5 through 1, then negatively phrased items must be coded 1 through 5.

An overall score is computed for each respondent by adding points for each item. Creating such a total assumes that the items are measuring the same things and that each item reflects equal elements of the characteristic being studied; that is, one item should not carry any more weight than the others.

Constructing a Likert scale requires more than just listing a group of statements. A large pool of items should be developed, usually 10 to 20, that reflect an equal number of both favorable and unfavorable attitudes. It is generally not necessary to include items that are intended to elicit neutral responses, because these will not help to distinguish respondents. The scale should be validated by performing item analyses that will indicate which items are truly discriminating between those with positive and those with negative attitudes. These items are then retained as the final version of the scale, and others are eliminated. If respondents are equally likely to agree with both favorable and unfavorable statements, then the scale is not providing a valid assessment of their feelings about a particular issue. The basis of the item analysis is that there should be correlation between an individual's total score and each item response. Those who score highest should also agree with positively worded statements, and those who obtain the lowest total scores should disagree. Those items that generate agreement from both those with high and low scores are probably irrelevant to the characteristic being studied, and should be omitted.

Semantic Differential

Attitudes have also been evaluated using a technique called the semantic differential.²⁰ This method tries to measure the individual's feelings about a particular object or concept based on a continuum that extends between two extreme opposites. For example, we could ask respondents to rate their feelings about natural childbirth by checking the space that reflects their attitude on the following scale:

The semantic differential is composed of a set of these scales, using pairs of words that reflect opposite feelings. Typically a 7-point scale is used, as just shown, with the middle representing a neutral position. This scale is different from the Likert scale in two ways. First, only the two extremes are labeled. Second, the continuum is not based on agree/disagree, but on opposite adjectives that should express the respondent's feelings about the concept. Figure 15.4 illustrates a semantic differential to explore self- image in a group of elderly women who reside in a nursing home.

Research has demonstrated that the adjective pairs used in this scale tend to fall along three underlying dimensions, which have been labeled evaluation, potency and activity.^20,21 Evaluation is associated with adjectives such as nice-awful, good-bad, clean-dirty, valuable-worthless and helpful-unhelpful. Some concepts that lie on the positive side of this dimension are doctor, family, peace, success and truth. Negative evaluation concepts include abortion, disease, war and failure. Potency ideas are big-little, powerful-powerless, strong-weak, large-small and deep-shallow. Strong potency concepts include bravery, duty, law, power and science. Negative concepts include baby, love and art. The activity dimension is characterized by fast-slow, alive-dead, noisy-quiet, young-old, active-passive and sharp-dull. Strong activity concepts are danger, anger, fire and child. Concepts that lie toward the negative activity side are calm, death, rest and sleep. The ratings shown in Figure 15.4 are labeled according to their respective dimensions. It is a good idea to mix up the order of presentation of the dimensions in listing the scales.

The semantic differential is scored by assigning values from 1 to 7 to each of the spaces within each adjective pair, with 1 representing the most negative response and 7 indicating the positive extreme. To avoid biases or a tendency to just check the same column in each scale, the order of negative and positive responses should be randomly varied. For instance, in Figure 15.4, ratings of weak-strong, slow-fast and ugly-beautiful place the negative value on the left; all other scales have the positive value on the left. A total score can be obtained by summing the scores for each rating. Lower total scores will reflect generally negative feelings toward the concept being assessed, and higher scores represent generally positive feelings. Statistical procedures, such as factor analysis, can be applied to the scale ratings to determine if the evaluation, potency, and activity ratings tend to go together (see Chapter 29 for a description of factor analysis). In this way, the instrument can be used to explore theoretical constructs.

Visual Analogue Scales

A visual analogue scale (VAS) is one of the simplest methods to assess the intensity of a subjective experience. A line is drawn, usually fixed at 100 mm in length, with word anchors on either end that represent extremes of the characteristic. The intermediate levels along the line are not defined. Respondents are asked to place a mark along the line corresponding to their perceived level for that characteristic. The VAS is scored by measuring the distance of the mark from the left-hand anchor in millimeters. This method has also been used to measure a variety of characteristics,²² most extensively for pain,^23,24 as shown in Figure 15.5. The VAS can be used to evaluate a variable at a given point in time or its degree of change over time.

The scores obtained with a VAS have generally been treated as ratio level data, measured in millimeters.^25,26,27 This assumption permits VAS scores to be added to obtain a mean and subjected to parametric statistical procedures. Some have argued that the scores are only psuedo ratio, and should be treated as ordinal, handled with nonparametric statistics.²⁸ They suggest that the individual marking the line is not truly able to appreciate the full continuum, evidenced by ceiling effects²⁹ and a tendency to cluster marks at certain points.³⁰ Therefore, even though the actual readings from the scale are obviously at the ratio level, the true measurement properties may be less precise. This dilemma will continue to emerge in studies using the VAS.³¹

The simple format of the VAS continues to make it a popular method for assessing unidimensional characteristics. This points out one disadvantage of the technique, however, in that each VAS is only capable of evaluating one dimension of a trait. Researchers often incorporate several VAS lines, each with different anchors, to assess related aspects of the characteristic being measured.^32,33

Cumulative Scales

In a summative scale, several item scores are added to create a total score. One of the limitations of this type of measure is that the total score can be interpreted in more than one way. Suppose we have a scale, scored from 0 to 100, that measures physical function, including elements related to locomotion, personal hygiene, dressing and feeding. Two individuals who achieve a score of 50 may have obtained this score for very different reasons. One may be able to walk, but is unable to perform the necessary upper extremity movements for self-care. Another may be in a wheelchair, but is able to take care of his personal needs. Therefore, a summed score can be ambiguous. This potential outcome reflects the fact that the items within the scale actually reflect different components or dimensions of the trait being measured, in this case physical function, which are not all equal.

Cumulative scales (also called Guttman scales) provide an alternative approach, wherein a set of statements is presented that reflects increasing intensities of the characteristic being measured. This technique is designed to ensure that there is only one dimension within a set of responses; that is, there is only one unique combination of responses that can achieve a particular score. For instance, in a cumulative scale a respondent who agrees with item 2 will also have had to agree with item 1; one who agrees with item 3 will have had to agree with items 1 and 2; and so on. Therefore, although there may be several combinations of responses that will result in a total score of 10 for a summative scale, there is only one way to achieve that score on a cumulative scale. Consider the following statements which were included in a self-assessment interview of elderly people concerning their functional health status.³⁴

1. I can go to the movies, church or visiting without help.

2. I can walk up and down to the second floor without help.

3. I can walk half a mile without help.

4. I am not limited in any activities.

5. I have no physical conditions or illnesses now.

6. I am still healthy enough to do heavy work around the house without help.

If these items represent a cumulative scale, then all those who can walk half a mile can also climb stairs to the second floor and go out visiting. Those who cannot walk half a mile should not be able to do heavy housework and probably have some limiting illness or physical condition. The development of this scale is, therefore, based on a theoretical premise that there is a hierarchy to this dimension of health.

Each item in the cumulative scale is scored as 1 = agree or 0 = disagree. A total cumulative score is then computed for all items. The maximum score will be equal to the number of items in the scale. A respondent who achieves a score of 2 would have had to agree only with items 1 and 2. If he agreed with items 1 and 3 only, the scale would be faulty because the set of statements would not constitute a hierarchy in terms of the characteristic being assessed. In reality, such scales are not free of error, and some of the subjects can be expected to present inconsistent patterns of response. In the analysis of the response categories for functional health, researchers found that most of their subjects could participate in social activities (86%) and that the fewest could do heavy work around the house, like shoveling snow and washing walls (21%).³⁴ The frequencies for other responses ranged between these two extremes, supporting the cumulative scale.

Rasch Analysis

The issues of hierarchical assessment extend to many of the questionnaire instruments that have been developed to assess functional and health outcomes. In most such scales, items are marked using ordinal values, and a total score is generated. For example, we could ask elderly patients if their health limits their function based on several ADL items, as follows:

Although this is an obviously abbreviated scale for the sake of example, a person who is independent in all three items would obtain a total score of 9. A person who is severely limited in all three tasks would receive a total score of 3. For this total score to be meaningful, however, three criteria must be met. First, the scale items must reflect a unidimensional construct. For instance, the ability to eat is not necessarily related to the ability to walk indoors or climb stairs; that is, these items may be part of different dimensions of function.³⁵ If so, the sum of scores on these items would not reflect a unified construct—sort of adding apples and oranges. Therefore, two patients who obtain a score of 5 may not demonstrate the same functional profile.

Second, the items must progress according to a hierarchical model from easy to difficult, so we can determine if someone has more or less of the trait. This would also mean that the order of difficulty for the items is consistent for all patients, and that the range of the scale incorporates the extremes.³⁶ Therefore, if our sample functional scale was properly arranged, eating would be easier than walking indoors, and walking indoors would be easier than climbing stairs for everyone.

Third, we need a scale that will allow us to measure change within or across patients. As we have noted before, ordinal values may present problems in this regard because they have limited sensitivity and precision. A patient might improve in his ability to climb stairs, but not enough to be scored at a higher level of independence. Therefore, for a score to be meaningful, units of measurement must have equal intervals along the scale, to account for magnitude of change. These objectives can be achieved using a technique called Rasch analysis, which statistically manipulates ordinal data to create a linear measure on an interval scale.^{37,38,39,40,41}

We can describe this process using items from the Functional Independence Measure (FIM), a popular instrument for assessing function in rehabilitation settings. The FIM is an 18-item scale designed to evaluate the amount of assistance a patient needs to accomplish activities of daily living (ADLs).⁴² The items measure both motor and cognitive functions. Each item is scored on an ordinal scale from 1 (total assist) to 7 (total independence). The larger the total score, the less assistance the patient requires. Theoretically, then, if the scale represents a singular construct of function, the total score should reflect an "amount" of independence; that is, we can think of individuals as being "more" or "less" independent.

The Rasch Model

Now let us conceive of function as a line, representing the continuum of function, as shown in Figure 15.6. For this example we will use the five items in the cognitive subscale, listed in Table 15.1.^∗ We construct the line using items in the scale, with easier items at the base, and harder items at the top. Using data from a patient sample and specialized computer programs,⁴³ the Rasch analysis determines the order of difficulty of the items, and locates them along this continuum, to show how they fit a unidimensional model of function. The analysis will also position patients along this line according to "how much" or "how little" cognitive function they have. The arrangement of items in Figure 15.6 illustrates these concepts based on a study by Heinemann et al⁴⁴ who performed a Rasch analysis on the motor and cognitive portions of the FIM. The figure illustrates two facets of this scale: On the right, the items are ranked in relation to their difficulty; and on the left, the patients are positioned relative to their abilities.⁴⁵ The more difficult items have a higher score, and patients who have these abilities (they are more independent) will also be placed near the top of the scale. Patients who are less functional are placed toward the bottom of the scale, as they are only able to complete the easier items. In Figure 15.6, patient 5 was only able to achieve the auditory comprehension and verbal expression items, while patients 1 and 2 were able to achieve all five items.

If the scale truly represents one functional construct, it should meet three measurement principles.^14,45 First, the total score on the scale should reflect level of function implied by the items; second, the items will range in difficulty; and third, the rank order of difficulty will not change from person to person. The results of the computerized Rasch analysis will show where each individual respondent fits along the continuum; the level of difficulty achieved by each item on an interval scale; and goodness-of-fit of the model, showing how well each item matches the cumulative scale.¹⁴

Measurement Criteria

Several criteria are used to judge the adequacy of a scale as part of a Rasch analysis:

1. Item difficulty refers to the position of items within the hierarchical scale. It is expressed as a logit^†, or log-odds unit, with a central zero point, allowing items to be scaled as positive or negative. The items are ordered so that the degree of function becomes systematically greater as the items become harder; that is, patients who have greater functional ability will "pass" the more difficult items. Therefore, it becomes possible to determine how close or far apart items are in difficulty, not just their rank order of difficulty. Ideally items are positioned equally across the scale, not leaving large gaps. As shown in Table 15.1 and Figure 15.6, the five items on the FIM range in difficulty from −0.45 to 0.53, with a reasonable spread of scores. The most difficult item is problem solving, and the easiest item is verbal expression. If gaps are identified, they suggest where items need to be added better to reflect the continuum.

2. Item fit is the extent to which the individual items conform to the unidimensional model. Person fit represents the extent to which individuals fit the model. The Rasch analysis develops a probability model that predicts what scores should be for each item and person. If we look at the continuum for the construct of cognitive function, for example, a good fit means that each item represents a level on the scale that will discriminate between those who require less assistance and those who require more assistance. Patients who are less functional will be placed toward the bottom, and those who are more functional will be placed toward the top; that is, the more functional individual will pass more of the items (and more of the difficult items), and the less functional individual will fail more of the items.

When expected relationships are not found, the responses are considered a misfit.^‡ For example, a Rasch analysis for the entire FIM scale has shown that combining all 18 items resulted in a large proportion of misfitting items;⁴⁴ that is, some of the more difficult items and more functional patients were not placed at the top (supporting the separation of motor and cognitive subscales).

Fit statistics are calculated for each item to reflect how well the items conform to the hierarchical model. These statistics are expressed as a mean square residual (MNSQ), which is the difference between the observed scores and the scores expected by the model. If the observed and expected values are the same, the MNSQ will equal 1.0.^§ Higher MNSQ values indicate greater discrepancy from the model; that is, the item is not consistent in its level of difficulty across patients.^** It would then be reasonable to consider revising the scale, either by eliminating the item or rewording it to remove ambiguity. If patients are misfit, the researcher must examine their characteristics, potentially identifying subgroups in the population. In the FIM study by Heinemann and colleagues,⁴⁴ several patient groups were evaluated, demonstrating that differently ordered cognitive scales were needed to represent groups with and without brain dysfunction. For instance, patients with right- and left-sided strokes did not demonstrate similar difficulty with verbal expression.

3. Item separation reflects the spread of items, and person separation represents the spread of individuals. Ideally, the analysis will show that items can be separated into at least three strata that represent low, medium and high difficulty,⁴⁸ although a good scale may actually delineate many strata to clarify the construct. Statistically, this spread is related to measurement error or reliability; that is, the more reliable a scale, the more likely the item or person score represents the true score. Measurement error should be small so that segments of the scale are separated by distances greater than their measurement error alone. Separation statistics may be expressed as a reliability coefficient, or the ratio of the sample standard deviation to the standard error of the test.³⁹ Conceptually, this is a ratio of the true spread of scores divided by the measurement error.

An understanding of measurement principles applied to questionnaires is essential if we want to use scores as part of our patient evaluations or to look at group performance over time. We must consider the potential for misinference when ordinal scales are used. Rasch Item Response Theory provides an important technique for testing our assumptions in clinical measurement. Several useful examples of Rasch analysis can be found in the literature.^{49,50,51,52,53}

The Q-sort is an analytic technique used to characterize attitudes, opinions or judgments of individuals through a process of comparative rank ordering.⁵⁴ The technique involves presenting an individual with a set of cards containing a series of written items such as statements, ideas, phrases or pictures. The individual is asked to sort the cards into piles according to some scaled criterion. For example, cards may list areas of clinical research in rehabilitation, and the subject may be asked to sort the cards according to high versus low priority. The criterion is defined on a discrete continuum, such as an 11-point scale, with 0 representing no interest at all and 10 representing the highest priority. Scales of different widths may be appropriate for different variables; however, a wide enough continuum is necessary to see a clear distribution.

The subject must sort through the cards and place them in piles representing each rank along the continuum; however, the researcher specifies how many cards are to go into each pile, so that the subject is faced with forced choices. For example, we could present a deck of 60 cards, each with a topic of clinical research, and ask a subject to form piles according to the distribution shown in Table 15.2. The subject would be instructed to read through the entire set of cards, and to place in Pile 0 the one card containing the single least important topic and in Pile 10 the single topic of highest priority. Then, from the remaining 58 cards, the subject would place the two least important items in Pile 1 and the two most important in Pile 9. This process continues for Piles 2 and 8, 3 and 7, and 4 and 6. The remaining 12 cards are placed in Pile 5, essentially a neutral pile. The subjects are free to replace or move any card to another pile at any time during the sorting procedure until they are satisfied with results. Although Q-distributions are essentially arbitrary, for statistical convenience the distribution is usually arranged to resemble a normal distribution, with fewer and fewer items toward the extremes. Although the number of cards used will vary according to the research question, Q-sorts generally range from a low of 60 to a high of 100 to 120 items. Too large a deck is difficult to sort through, and too small a deck will not provide sufficient stability for statistical reliability.

Q-methodology provides an empirical basis for exploring abstract ideas and theories, generally with good reliability. It can be applied to a variety of research questions and is quite flexible. It is possible to use the technique to answer questions that require the use of two or more related sets of items, or a single set of items can be sorted on more than one scale. For instance, Biddle et al⁵⁵ used a Q-sort to establish key characteristics of effective primary care training experiences for third-year medical students. The students completed a Q-sort using three sets of items: preceptor characteristics, site characteristics and a combination of the two.

The approach to analysis of Q-sort data will depend on the research question. For some purposes, descriptive statistics, such as averages, percentages, and simple tallies of rank orderings, will be sufficient. For example, Q-sort has been used to rank applicants to a physical therapy program.⁵⁶ More complex statistical procedures can also be applied to the Q-methodology. Correlations are often used to determine if the sorts of several subjects are related, usually using a nonparametric procedure for correlating ranks. For instance, Kovach and associates⁵⁷ studied employees in long-term care facilities to describe factors that facilitate positive change in the care of patients with dementia. The employees were asked to rank personal factors and facility factors. Using Spearman correlation coefficients, the researchers found little congruence between real and ideal facility characteristics, but a strong relationship between real and ideal personal characteristics. Factor analysis and content analysis are also used to uncover underlying themes in the Q-sort. For example, in the study of medical students' perceptions of clinical experiences, Biddle et al⁵⁵ use content analysis to group responses into six categories, including patients, staff characteristics, preceptor's personal characteristics, programmatic issues, educational opportunities, and the strongest theme around preceptor teaching characteristics.

The Q-sort is limited in its generalizability because subjects are not randomly chosen. Samples tend to be small because of the logistic difficulties administering the technique. Replication of Q-sorts over many samples is necessary to demonstrate validity of findings. Use of already established Q-sorts (sets of items) is helpful to validate findings by replication. When Q-sorts do not exist, the researcher must establish the content validity of items used.

Many questions of interest in medical and social sciences are related to practice, values or standards, and are best answered by developing consensus around a specific issue. In a Delphi survey, a panel of experts is asked to complete a series of questionnaires to identify their opinions.^58,59 The Delphi technique differs from typical questionnaires in several ways. The most distinguishing difference is the use of several rounds of questionnaires, typically two or three. In each round, the researcher reviews and collates the results, and then distributes these findings to the panel for their response. This process generally continues until the responses are consistent with the previous round, demonstrating consensus.

This technique was used, for example, in a study of opinions of physical therapist managers on the importance of specific knowledge and skills in leadership, administration, management and professionalism (LAMP).⁶⁰ In a first round, the panel was asked to respond to a list of LAMP components, modifying or adding items. Data from round one were compiled by a research team, resulting in 178 items. In the second round, panel members were asked to rank the importance of each item on a Likert scale. The purpose of the third and final round was to reach consensus on the level of knowledge and skills needed by a new graduate. The panel agreed that 44% of the items were important at a high to moderate level of skill, and that no skill was needed in 29% of the items. These findings present useful information for the design of professional curricula.

The Delphi survey has great potential for planning and problem solving for a variety of practice issues. Investigators have used this technique to establish consensus on indicators for assessing the severity of rheumatoid arthritis through medical records,⁶¹ quality indicators for care of general medical conditions in nursing home residents,⁶² clinically important differences in the evaluation of change in health-related quality of life for patients with asthma⁶³ and effective smoking prevention strategies for female adolescents.⁶⁴ It is an efficient method because the members of the panel do not need to come together, making large response groups feasible, including individuals at a distance. Consensus is developed without interaction among respondents, avoiding the potential for group biases, such as one dominant individual swaying others in the group. Responses are shared without any one individual being challenged by the group. The anonymity offered by this method will also encourage honest responses from the panel. The disadvantages include the cost of printing and mailing, and the need to maintain a commitment by the panel members over several rounds. Researchers will generally use follow-up reminders to encourage full participation.

The first step in the analysis of data is to collate responses and enter them into a computer. Each item on the survey is a data point, and must be given a variable name, often the item number. The researcher must sort through each questionnaire as it is returned, or through all responses from an interview, to determine if responses are valid. In many instances, the respondent will have incorrectly filled out the survey, and that respondent may have to be eliminated from the analysis. Some questions may have to be eliminated from individual questionnaires because they were answered incorrectly, such as putting two answers in for a question that asked for a single response. The researcher must keep track of all unusable questionnaires to report this percentage in the final report.

Responses to closed-ended questions are coded; that is, responses are given numeric codes that provide labels for data entry and analysis. For instance, sex can be coded 0 = male, 1 = female. We could code hospital size as 1 = less than 50 beds, 2 = 50 to 100 beds, and 3 = over 100 beds. These codes are entered into the computer to identify responses. Using codes, the researcher can easily obtain frequency counts and percentages for each question, to determine how many subjects checked each response.

The analysis of survey data may take many forms. Most often, descriptive statistics are used to summarize responses. When quantitative data such as age are collected, the researcher will usually present averages. With categorical data, the researcher reports the frequency of responses to specific questions. These frequencies are typically converted to a percentage of the total sample. For example, a researcher might report that 30% of the sample was male and 70% was female, or in a question about opinions, that 31% strongly agree, 20% agree, 5% disagree, and so on. Percentages should always be accompanied by reference to the total sample size, so that the reader can determine the actual number of responses in each category. Percentages are usually more meaningful than actual frequencies because sample sizes may differ greatly among studies.

Another common approach to data analysis involves the description of relationships between two or more sets of responses. For instance, a survey might contain a question asking a person's gender, and other questions might ask about attitudes toward abortion. The researcher can then examine the frequency of responses to the attitude questions in relation to each respondent's sex. Cross-tabulations are usually presented, showing the number of males and females who answered positively or negatively to each attitude question. The chi-square test (χ²) can be used to examine this relationship statistically, to determine if there is a significant relationship between the two variables (see Chapter 25).

When questionnaires include a scale, researchers may want to look at sums as a reflection of the respondents' answers. These sums may be presented for the entire scale, or subscales may be analyzed. Measurement properties of such scales must be considered, such as the potential need for weighting items differently within a scale, or the decision on which items belong to a subscale.

Depending on the length and complexity of the questionnaire, researchers may present response percentages for all questions on the survey, or they may simply summarize the more important relationships that were studied. Some reports present purely narrative descriptions of the results; others include tables showing the responses to each question. The author must determine which type of presentation will be most effective for the data.

If a questionnaire is developed for the purpose of assessing a particular characteristic, such as function or pain, the researcher should evaluate the reliability and validity of the instrument. Different types of reliability and validity should be tested, depending on the type of questions and purpose of the assessment (see Chapters 5 and 6). The examination of an instrument's measurement properties is essential if the instrument will be used by others.

Even though interviews or questionnaires do not require physical interaction with subjects, these studies must go through a formal process of review and approval by an Institutional Review Board (see Chapter 3). Researchers must be able to demonstrate the protection of subjects from psychological risk and the guarantee of confidentiality. The IRB will want assurances that all relevant individuals have been notified and are in support of the project. Surveys will often receive expedited reviews by a review board.

Individuals who participate in face-to-face interviews can be given an informed consent form to sign in the presence of the interviewer and a witness. Consent to participate in telephone interviews is implied by the individual's participation. The researcher is obliged to give telephone respondents full information at the beginning of the call, so that they can decide whether they want to continue with the interview. Similarly, consent for mail questionnaires is implied by the return of the questionnaire. The cover letter provides the information needed.