A Prevention 101 Series Publication Methods for Assessing College Student Use of Alcohol and Other Drugs The Higher Education Center for Alcohol and Other Drug Abuse and Violence Prevention Funded by the U.S. Department of Education This publication was funded by the Office of Safe and Drug-Free Schools at the U.S. Department of Education under contract number ED-04-CO-0137 with Education Development Center, Inc. The contracting officer's representative was Richard Lucey, Jr. The content of this publication does not necessarily reflect the views or policies of the U.S. Department of Education, nor does the mention of trade names, commercial products, or organizations imply endorsement by the U.S. government. This publication also contains hyperlinks and URLs for information created and maintained by private organizations. This information is provided for the reader's convenience. The U.S. Department of Education is not responsible for controlling or guaranteeing the accuracy, relevance, timeliness, or completeness of this outside information. Further, the inclusion of information or a hyperlink or URL does not reflect the importance of the organization, nor is it intended to endorse any views expressed or products or services offered. U.S. Department of Education Margaret Spellings Secretary Office of Safe and Drug-Free Schools Deborah A. Price Assistant Deputy Secretary December 2008 This publication is in the public domain. Authorization to reproduce it in whole or in part is granted. While permission to reprint this publication is not necessary, the citation should be: U.S. Department of Education, Office of Safe and Drug-Free Schools, Higher Education Center for Alcohol and Other Drug Abuse and Violence Prevention, Methods for Assessing College Student Use of Alcohol and Other Drugs, Washington, D.C., 2008. To order copies of this publication, write to: The Higher Education Center for Alcohol and Other Drug Abuse and Violence Prevention Education Development Center, Inc. 55 Chapel Street Newton, MA 02458-1060 or call: 1-800-676-1730; TDD Relay-friendly, Dial 711 or fax: 617-928-1537 or e-mail: HigherEdCtr@edc.org This publication and other resources are available on the Web site for the U.S. Department of Education's Higher Education Center for Alcohol and Other Drug Abuse and Violence Prevention: http://www.higheredcenter.org. Methods for Assessing College Student Use of Alcohol and Other Drugs by William DeJong, Ph.D. Introduction If they are to develop effective programs and policies to reduce alcohol- and other drug-related (AOD) problems on campus, college and university officials must understand the nature and extent of these problems at their institution. Administrators can achieve this understanding only if they have accurate data on patterns of student substance use and related risk behavior. While focus groups, one-on-one interviews, and consultations with faculty and staff can be helpful, the best way to obtain these data is to conduct an annual survey using a randomly selected sample of student respondents. Unfortunately, many campus officials are unfamiliar with survey methods and do not have the time or staff to develop a sound data collection plan. As a result, they may avoid conducting routine campus surveys or implement them in ways that violate basic survey principles. Other administrators steer clear of student surveys due to fear of negative publicity if the findings were to be released to the public. Avoidance is not an option. Years of litigation over the concept of "reasonable care" in higher education law have shown that the courts will typically protect an institution that shows good faith by seeking to apply state-of-the-art prevention methods and evaluating their operation and effect.2 Currently, the state of the art in campus-based prevention is defined by the report A Call to Action: Changing the Culture of Drinking at U.S. Colleges, which reviews the research literature on successful prevention strategies and provides recommendations for best practices. A key recommendation in that report is to "integrate research into college alcohol program planning," including "data describing the dimensions of the college drinking epidemic and its effects on students."3 This guide offers a straightforward method for gathering and reporting student survey data on substance use-related problems. It will be of particular interest to program directors for AOD prevention programs on campus, or to members of a campus-based task force or campus and community coalition that is charged with assessing the need for new prevention programs and policies. Senior campus officials also will find this guide useful for gaining a basic understanding of methodology for on-campus surveys that focus on campus violence and other health and safety issues. __________________ Sidebar: The Drug and Alcohol Abuse Prevention Regulations (Education Department General Administrative Regulations, Part 86) require institutions of higher education to conduct a biennial review on the effectiveness of their alcohol and other drug programs and the consistency with which sanctions are applied for policy infractions.1 A student survey can be an important component of this biennial review or other evaluations of campus-based prevention programs and policies. __________________ Questionnaire Design Good survey instructions, located either on the questionnaire's front page or in a cover letter, will introduce the scope and purpose of the survey using neutral language to avoid biasing how respondents complete the survey. For example: "This survey is about alcohol-related attitudes, norms, and behavior. Your answers will be used to learn more about college students and their perceptions of other students' alcohol use." The instructions should explain the benefits and risks of participation, describe any incentives being offered, and clarify how the survey will protect respondents' anonymity or confidentiality. (Appropriate procedures are explained on page 7.) Also, the instructions should state that the students' participation is voluntary, and that they can skip any questions that make them uncomfortable. Providing contact information for the head of the research group is another good practice, so that students can follow up with any questions or concerns they might have. A well-designed survey questionnaire will ask a wide range of questions to determine the scope of AOD use and its consequences, identify underlying causes, and assess current prevention programs and policies. Listed here are the major content areas that a survey can cover: * Personal characteristics that may be associated with AOD use, including gender, age, racial and ethnic background, relationship status, and current employment * Family characteristics, including parents' education level and substance use histories * Academic standing, including year in school, full- or part-time status, grade point average * Current residence, including living situation (alone, with roommates, with family), location of residence (on vs. off campus), and type of residence (fraternity or sorority house, residence hall or dormitory, house or apartment, or other) * Participation in various student activities (e.g., community service, religious group, fraternity or sorority, intercollegiate athletics) * Annual and 30-day prevalence of alcohol, tobacco, and other drug use * Levels of recent alcohol use (see "Example Measures of Alcohol Use") * Alcohol use in specific contexts (defined by occasions and settings) * Sources of alcohol (e.g., retail outlets, acquaintances) * Perceived benefits and risks of alcohol use * Consequences due to own alcohol use (e.g., hangover, had unprotected sex) * Consequences due to other students' alcohol use (e.g., interrupted sleep, personal property damaged) * Strategies used by the student to avoid heavy drinking (e.g., pacing alcohol consumption) * Strategies used by the student to avoid driving after drinking (e.g., designated driver) * Perceptions of campus drinking norms * Awareness and support of campus substance abuse policies, local laws, and their enforcement (see "Questions to Assess Student Support for Policy") * Perceptions of the campus and community environment (e.g., ease of alcohol access, availability of extracurricular and recreational options, level of policy and law enforcement, promotion of a strong nonuse message for students under the minimum legal drinking age) __________________ Sidebar: Example Measures of Alcohol Use A "drink" is defined as "a bottle of beer (12 oz.), a glass of wine (4 oz.), a wine cooler (12 oz.), or a shot of liquor (1 oz.) served straight or in a mixed drink."a 30-Day Prevalence During the past 30 days, on how many days did you have at least one drink of alcohol? Weekly Consumption What is the average number of drinks you consume in a week? Heavy, Episodic Drinkingb Think back over the last two weeks. How many times have you had five or more drinks at a sitting? Peak Consumptionc Think back over the last two weeks. What was the greatest number of drinks you consumed at one sitting? For how many hours did you drink? __________ a Source: The Core Alcohol and Drug Survey (Long Form), Core Institute, Southern Illinois University Carbondale, Ill. (see http://www.siu.edu/~coreinst). b Source: The Core Alcohol and Drug Survey (Long Form). The survey should be timed so that it does not coincide with a vacation break or the subsequent two- week time period, as students' drinking behavior during vacation may be different than when classes are in session. c Source: Social Norms Marketing Research Project, Education Development Center, Inc., Newton, Mass. (see http://www2.edc.org/snmrp/survey.pdf). When analyzing the survey data, this information can be combined with the respondent's gender and body weight to estimate the maximum blood alcohol concentration (BAC) reached during the drinking episode.4 __________________ Sidebar: Questions to Assess Student Support for Policy To what extent do you support or oppose the following possible policies or procedures? [Response alternatives: strongly support, support, oppose, strongly oppose] Examples: * Use stricter disciplinary sanctions for students who engage in alcohol- related violence. * Apply stricter penalties for the use of false IDs to purchase alcohol illegally. * Prohibit kegs on campus. * Eliminate low-price bar and liquor store promotions targeted to college students. ____________ Source: DeJong, W.; Towvim, L. G.; and Schneider, S. K. "Support for Alcohol Control Policies and Enforcement Strategies Among U.S. College Students at 4- Year Institutions." Journal of American College Health 56: 231-236, 2007. __________________ Questions designed to assess specific programs and policies also can be added. Several survey instruments are available that measure these content areas. It is wise to select one of these whenever possible, as they have been tested and used with other AOD programs: * American College Health Association (ACHA). ACHA offers the National College Health Assessment (NCHA). This online instrument broadly covers several topic areas: alcohol, tobacco, and other drug use; sexual health; weight, nutrition, and exercise; mental health; and personal safety and violence. Both paper and online survey administration is available. Users are able to compare their data against the NCHA's national survey database (see http:// www.acha- ncha.org/index.html). Users are charged a fee. * Center for College Health and Safety (CCHS). Based at Education Development Center, Inc., Newton, Mass., CCHS developed the Survey of College Alcohol Norms and Behavior for the Social Norms Marketing Research Project, a national research study to evaluate the effectiveness of social norms marketing in reducing high-risk drinking among college students.5 The survey, which can be used at no charge, includes innovative scales to measure student support for policy and perceived social capital on campus (see http://www2.edc.org/ snmrp/survey.pdf). * Core Institute at Southern Illinois University. The Core Institute markets several fee-for-service surveys. The best known is the Core Alcohol and Drug Survey, which has both a long and short form. Both paper and online survey administration is available. Another popular survey is the Campus Survey of Alcohol and Other Drug Norms, which includes many questions not available in the other instrument. The Core Institute has also developed a faculty and staff survey that may be of interest to campus administrators (see http:// www.siu.edu/~coreinst). Some campuses administer a general health survey one year and an AOD-specific survey the next. Campus administrators may want to develop additional questions to ask about issues specific to their campus. Several available resources describe the process of question development and offer guidance on question formatting and phrasing.6 Deciding how extensively to cover each topic area, while keeping the survey instrument at a manageable length, can be a challenge. Most texts on survey design avoid specifying a maximum survey length. The standard instruments listed above can be completed within 30 minutes by most students. Within that time frame, the longer the survey, the greater the incentives needed to get a good response rate. Pretesting will be necessary to assess how long the survey takes and whether the incentives being offered are sufficient. Conducting a Survey The remainder of this guide outlines how to implement and analyze a mail or online survey of student AOD use. The major steps are: (1) obtaining institutional review board (IRB) approval, (2) selecting the study sample, (3) administering the survey, (4) analyzing the data, and (5) reporting the results. Using this method, campus administrators will have a first-rate study-one that surveys a random sample of students and achieves an acceptable response rate. Alternative methods for survey administration, during course registration or during classes, are also described. Obtaining Institutional Review Board (IRB) Approval In nearly all cases, research involving human subjects must be reviewed by the college or university's institutional review board to ensure that the necessary steps have been taken to protect subjects from harm and to ensure the anonymity or confidentiality of their responses. Approval from the IRB is needed for both the survey instrument and the method of administration. If an institution has any federally funded research and does not have an IRB, one will need to be constituted. To find out what is required, contact the U.S. Department of Education, Office of the Chief Financial Officer (see http://www.ed.gov/about/offices/list/ocfo/humansub.html) or the U.S. Department of Health and Human Services, Office for Human Research Protections (see http://www.hhs.gov/ohrp). Key issues that an IRB will consider include whether (1) data will be collected or stored in a manner that allows the identification of any respondents; (2) student participation is voluntary; (3) any questions may cause emotional discomfort or psychological stress, and (4) the benefits to participants outweigh the risks. Participant benefits might include the opportunity to reflect on their AOD use, which might lead to positive changes in AOD-related knowledge, attitudes, and behaviors. Note that federal regulations prohibit payment or other compensation from being considered a participant "benefit" when doing a risk or benefit analysis. The IRB also will consider benefits of the research for the campus, and even for society as a whole. Selecting the Study Sample In conducting student surveys, many colleges choose a convenience sample, polling only those students who can be contacted easily through courses, on- campus organizations, or during registration. This method is described in a later section. A preferred method is to select a truly random sample. The advantage of administering a random-sample survey-namely, that the survey results are known to be representative of the student body as a whole-makes the extra work involved well worth the effort. This section reviews the steps in selecting an appropriate sample. The first step is to find a means of identifying all potential respondents (what researchers call the "sampling frame"). In nearly all cases, the institution's official enrollment list, which should be available from the college registrar, can serve this purpose. Having an up-to-date student list with accurate mailing addresses and telephone numbers is essential. The list's accuracy should be assessed after the fact. The Social Norms Marketing Research Project, which dropped less than 4 percent of its national sample, deleted names from the sample when (1) there were two or more pieces of undeliverable mail; (2) telephone contact revealed that the student was not enrolled in the institution, had graduated, or was spending the semester abroad; (3) the registrar verified that nonresponding students unreachable by telephone were not enrolled after all; or (4) respondents indicated on the survey that they were not enrolled or were in graduate or professional school.7 The second step is to pick a random sample of students from the list. When an institution has a computer-based student records system, administrators will be able to use a computer program to generate the random sample. Absent this capability, the administrator can use a random number table to select the first name on the enrollment list and then select every Xth student down the list to be in the sample. With this procedure, it is best if the names on the enrollment list are randomly ordered, rather than alphabetized. The size of X will depend on the total enrollment and the desired sample size. For example, to draw a sample of 500 students from an enrollment list of 20,000 undergraduates, administrators would use a random number table to pick a student from the first 40 names (20,000 divided by 500) and then add every 40th name on the list to the sample. Determining the Sample Size How large a random sample should be surveyed? There are three deciding factors: (1) the level of precision that campus administrators want their survey to have, (2) the degree of confidence they want to have in its results (the estimates), and (3) the size of the college's student body. To illustrate, imagine that researchers at a university with 10,000 students have received surveys from 964 randomly selected students, and that from this survey the researchers estimate that 56 percent of the university's students can be classified as heavy drinkers. It is highly unlikely that this estimate is exactly correct. More likely, the true level of heavy drinking among all 10,000 students is somewhat higher or lower than 56 percent. How much higher or lower? It turns out that, with a sample size of 964, the researchers can be 95 percent confident that the true level of heavy drinking is within 3 percent of the survey estimateÑthat is, somewhere between 53 and 59 percent. This means that there is only a 5 percent chance that the actual level of heavy drinking is either higher or lower than this predicted range-that is, lower than 53 percent or higher than 59 percent. In this scenario, the researchers would say that the 95 percent level of confidence is plus or minus 3 percent, and that the 95 percent confidence interval is 53 to 59 percent. If the researchers at this institution would have been satisfied with a level of precision of 5 percent, rather than 3 percent, then they could have gotten by with a sample of only 370 completed surveys. In this case, the 95 percent confidence interval would have been 51 to 61 percent (i.e., 56 percent, plus or minus 5 percent). The degree of precision is especially critical when comparing the results of two surveys. All other things being equal, a higher degree of precision (say, plus or minus 3 percent, rather than plus or minus 5 percent) increases the likelihood that a measured difference in outcome between two surveys will be declared to be statistically significant. Return again to the original situation, in which the researchers were working with a level of precision of 3 percent and a range estimate of 53 to 59 percent. This time, however, imagine that the researchers wanted to be 99 percent confident, rather than 95 percent confident, in this range estimate. This degree of confidence would require a sample of 1,557 completed surveys. In this case, there would be only a 1 percent chance that the actual level of heavy drinking is either higher or lower than the predicted range (i.e., the 99 percent confidence interval) of 53 to 59 percent. Table 1 shows the relationship between the level of precision specified for a percentage estimate (e.g., plus or minus 3 percent), the confidence limit (95 percent or 99 percent), the size of the college's student body, and the required sample size of completed surveys. What is immediately obvious from the table is that as the level of precision moves from 5 to 3 percent, and as the confidence limit moves from 95 to 99 percent, the required sample size increases dramatically. As can be seen in the table, in general, administrators should secure completed surveys from a sample of between 400 and 1,000 students. If the college's budget permits it, a sample of 1,000 is far superior. Even under the severest budget constraints, the sample should not be any smaller than 200. __________________ TABLE 1. Random sample sizes needed to achieve various degrees of precision for percentage estimates from student surveys Degree of Precision a 3% (d = .03) Student Body Size 500 Confidence Limit b 341 (95%) 393 (99%) Student Body Size 1,000 Confidence Limit b 516 (95% 649 (99%) Student Body Size 2,000 Confidence Limit b 696c (95%) 959 (99%) Student Body Size 5,000 Confidence Limit b 800 (95%) 1,347 (99%) Student Body Size 10,000 Confidence Limit b 964 (95%) 1,557 (99%) Student Body Size 20,000 Confidence Limit b 1,013 (95%) 1,688 (99%) Degree of Precision 5% (d = .05) Student Body Size 500 Confidence Limit b 217 (95%) 285 (99%) Student Body Size 1,000 Confidence Limit b 278 (95%) 399 (99%) Student Body Size 2,000 Confidence Limit b 322 (95%) 498 (99%) Student Body Size 5,000 Confidence Limit b 357 (95%) 586 (99%) Student Body Size 10,000 Confidence Limit b 370 (95%) 622 (99%) Student Body Size 20,000 Confidence Limit b 377 (95%) 642 (99%) a The degree of precision, also known as the margin of error, is the amount of tolerated error (e.g., plus or minus 3 percent) in a percentage estimate from a student survey (e.g., 48 percent of the students are heavy drinkers). b The confidence limit is the relative degree of certainty with which a percentage estimate is made (e.g., 95 percent confidence). c Example: For a percentage estimate that is accurate to within plus or minus 3 percent, and to be 95 percent confident in that estimate, a sample size of 696 is required if the total population size is 2,000. In this case, a finding from the student survey might be expressed as follows: "An estimated 48 percent of the students are heavy drinkers, with a 95 percent confidence limit of 45-51 percent." This table assumes that the estimated heavy drinking rate from the student survey will be 50 percent-or, equivalently, that the proportion estimate p will be 0.50. If there is no information about what the proportion is likely to be, then 0.50 is the most conservative estimate, as it dictates the largest possible sample size for the specified degree of precision and the chosen confidence limit. If a college has conducted a prior survey, then its value for the proportion estimate (p) can be used in the formula shown below to calculate what sample size is needed in the next survey. This formula can also be used when an institutionÕs actual student body size differs significantly from the precise values shown in the table: Where: n = the sample size needed; t = 1.960 for a 95% confidence limit, or 2.576 for a 99% confidence limit; p = the proportion estimate (e.g., .50); q = 1-p; d = desired degree of precision (e.g., for 3%, d = .03); __________________ In drawing the sample, allow for the anticipated response rate. As outlined below, there are several steps that can be taken to increase the percentage of students who complete the survey, but it is unlikely that the response rate will exceed 50 to 60 percent. Also allow for inaccuracies in the college's enrollment list; a good rule of thumb is to add up to 20 percent to the desired sample size. To illustrate, imagine that administrators wanted a final usable sample of 1,000 respondents. Assuming a 50 percent response rate, they would draw an initial sample of 2,000 students. Then, to account for errors in the enrollment list, they would increase the initial sample by 20 percent to 2,400 students. Thinking along these lines, campus officials who expect a response rate well below 50 percent may be tempted to compensate by administering the survey to all or a very large number of students. Actually, to produce a truly representative sample, it would be far better to select a smaller random sample and then offer incentives and do intensive follow-up work to get a higher response rate. How large a sample is drawn also depends on the kinds of comparisons that administrators might want to make among subgroups of students. The more complicated the analysis, the larger the sample needs to be. To illustrate why this is so, imagine that administrators surveyed a sample of 400 students, 50 percent of whom are male. Imagine further that 50 percent of the students, both males and females, are black and that 50 percent are white. This means that one- fourth of the students are black men, one-fourth are white men, and so forth. The size of each subgroup is only 100 students. This means that percentage estimates for any one of these subgroups would have a level of precision of only about 10 percent. If these four subgroups were subdivided even further-say by fraternity or sorority membership-then the level of precision would be even less. A good rule of thumb is to avoid subdividing the sample into categories beyond the point at which the number of students in any one of the resulting subgroups falls below 50. On the other hand, if comparing complexly defined subgroups is an important part of the study, then drawing a random sample of more than 1,000 students will avoid the pitfall of having subgroups that are too small. Finally, administrators always have the option of giving the survey to all of their students. Whether this is practical depends on the size of the student body and the institution's capacity to administer the survey in a cost-effective way. For example, if the college has walk-through course registration, students could be asked to complete a survey as part of that procedure, with due care taken to achieve high response rates while also maintaining the anonymity or confidentiality of student responses. Administrators at colleges with 1,000 or fewer students should give this option serious consideration. __________________ Sidebar: Telephone surveys of college students are not recommended. First, an adequate sampling frame is usually not available, because the registrar's office seldom has a record of student cell phone numbers. Second, due to frequent telemarketing calls, many people, including students, let voice-mail or an answering machine take their calls, or they use caller ID to screen out calls from unfamiliar numbers. And, once reached, people are now more likely to refuse to participate in telephone surveys than was the case several years ago.8 __________________ Administering the Survey: Mail and Online Surveys There are two main methods of survey administration: by mail and online. Until recently, most surveys were done by mail, but more and more colleges are switching over to Internet-based surveys. Such surveys offer considerable advantages, including cost savings, especially for large student samples; faster response times; improved survey design features such as forced-choice fields, programmed skip patterns, and interactive data validation; and automatic data entry into a database, ready for statistical analysis. The available evidence suggests that mail and online surveys yield data of comparable quality. In one randomized control study, the results showed no statistically significant differences in student demographics, response rates, or item completion errors between mail and online surveys of health risk behavi- ors.9 Another investigation found that reliability and validity assessments of alcohol use measures did not vary by online vs. traditional paper-based methods.10 Internet-based surveys are particularly appropriate for a college student population, the vast majority of whom use the Internet and e-mail. A 2002 study found that 86 percent of college students had gone online and that 72 percent check e-mail at least once daily.11 A growing number of colleges use e-mail and the Internet in course work,12 and many require students to check their campus e-mail accounts regularly for institutional communications. Some colleges and universities are unable to do online surveys because too few students use their college e-mail address, and campus officials have not yet required students to report their operative address. Obtaining Informed Consent. With a mail survey, students implicitly grant their consent to participate in the study after reading an introduction that explains the study's purpose, the voluntary and confidential nature of the survey, measures designed to protect anonymity or confidentiality, the set of offered incentives, the benefits and risks of participation, and contact information for the lead researcher. The introduction also makes clear that they can skip questions without penalty. For online surveys, students are sent an e-mail that includes a link for the survey's Web site address. When students click on the link, they should first see the study introduction. To encourage students to read this material, there can be a series of check boxes that the students click on as they read each paragraph. At the end, there should be a button that students click to indicate that they have read the material, understand it, and are willing to participate in the survey under the outlined conditions. Note that student responses are exported to a database only after the student clicks a "next" button at the end of each survey page, meaning that they may choose to terminate their participation at any time during the survey. Maximizing the Response Rate. A key concern for both mail and online surveys is the possibility of a low response rate, which would compromise the usefulness of the data. Strictly speaking, failing to obtain data from any student who is part of the random sample would make the data less representative of the entire student body. At one time, researchers conducting student surveys would look to obtain response rates of at least 70 percent. Recently, however, national mail surveys have achieved rates only between 50 and 60 percent, even with incentives and multiple reminder messages.13 While some researchers have reported low response rates with Internet-based surveys, others report achieving rates comparable to or even somewhat higher than mailed surveys.14 A key to achieving a response rate of 50 percent or more is to inform students about the steps being taken to ensure their anonymity or confidentiality. Research has established that self-report surveys about substance use are generally both valid and reliable, but only under conditions that convince students that their responses cannot be linked to them by anyone outside the research team.15 A paper-and-pencil survey can be completed anonymously. There should be no ID number or other identifying information on the questionnaire or, in the case of a mail survey, the return mailing envelope. Also, the instructions should remind students not to put their name on the questionnaire. For a mail survey, students can be asked to mail in a separate, self-addressed postcard with a unique ID number when they have completed the survey so that researchers can keep track of which students need to be sent future reminders. In practice, very few students return the postcard but not the survey. An online survey can be completed under conditions that maintain confidentiality, but not anonymity. It requires a more elaborate procedure. Each student selected to be in the study should be assigned a unique password ID, which must be entered to gain access to the online questionnaire. This is necessary so that students who are not selected cannot access the site, and to block multiple responses from the same student. If students begin the survey but do not complete it, they should be able to log in again, but their prior responses should not appear on the screen, thereby protecting students from the possibility of someone else finding their password ID and accessing their responses. Students should be informed that only their password ID, not their names or e- mail addresses, will be directly linked to their individual responses. The following steps can be taken to maintain students' confidentiality: (1) Encrypt all data so that it cannot be accessed by anyone other than a member of the research team. (2) Ensure that the data file will be available only to a predefined subset of research staff with password-guarded access. (3) Assign user privileges to other members of the research team that allow them only to view aggregate data, but not individual survey responses. (4) Maintain students' names and their unique password IDs in one database, while keeping the survey data, organized by student password ID, in a separate database with strictly controlled access. With this procedure, student names will never be directly linked to student data in any database, and only a small set of authorized users will have password-guarded access to both databases. Offering incentives is another key strategy for increasing the response rate.16 For a mailed survey, the Social Norms Marketing Research Project included a $1 bill with the first survey mailing as an up-front incentive.17 Students completing the survey, as indicated by a separately mailed return postcard with their ID number, became eligible for three prize drawings: a $100 cash prize per institution for students responding within one week; five $50 cash prizes per institution for students responding by the end of the semester; and a $1,000 national grand prize for students responding by the end of the semester.18 Smaller cash awards (or even prizes such as T-shirts, store coupons, and so forth) might work for a survey done on a single campus. Students completing an online survey can receive store discounts or coupons online. An important step is to send reminder messages to encourage students to complete the survey. Without follow-up, a random sample survey of students will typically have a 30 percent response rate. The Social Norms Marketing Research Project used the following schedule of mailings to boost response rates for its annual mail survey at 18 colleges and universities: (1) "teaser" postcard announcing the survey two to three days before the first survey mailing; (2) first survey mailing, sent about three weeks after the start of each institution's semester; (3) reminder postcard sent to nonrespondents; (4) second survey mailing, sent to arrive two weeks before spring break; (5) reminder postcard sent to remaining nonrespondents; (6) third survey mailing, sent about three weeks after spring break; and (7) fourth survey mailing, with a two-page version of the survey. Research staff even made reminder telephone calls after the second and third survey mailings; for each round, the caller left a voice message on the third attempt.19 Reminders for online surveys are simpler to execute. Some experts recommend sending a sequence of reminders, beginning three days after the initial e-mail notification and continuing episodically for one month.20 The number of follow- up reminders can vary, depending on the speed with which the desired response rate is obtained. Researchers have identified still other measures to boost response rates. As already noted, the survey introduction should inform students that their participation is voluntary and that they do not have to answer any question that makes them feel uncomfortable. It may also help to tell students that they are part of a small, randomly chosen group, and to ask for their prompt response, noting that this would help ensure that the survey results truly represent the student body. Finally, it is important to tell students that the college's IRB is requiring the researchers to conduct the survey in a manner that will protect the students' anonymity or confidentiality; describe the measures in place to meet that requirement; and provide the name, telephone number, and e-mail address of the IRB coordinator who can answer any questions they might have. Alternative Survey Methods The survey method just presented, which involves selecting a random sample of students and then surveying them by mail or online, is the best procedure for obtaining findings that are truly representative of the student body as a whole (unless, of course, it is possible to survey the entire student body). Unfor- tunately, not all colleges and universities are in a position to administer their surveys in this way. The most common barrier that institutions encounter is the time and money it takes to mail out the survey and then to do the necessary follow-up by mail or telephone to get a high response rate. And it must be emphasized that getting that high response rate is essential to the success of this method. Without it, administering the survey in this way may hold no special advantages over other, more convenient methods. Two alternative methods are described here: administering the survey during course registration or during classes. In practice, neither of these methods can result in a true random sample being selected, but they can produce response rates of 70 percent or even higher. To make these alternative methods acceptable, steps must be taken to ensure that the student sample is selected in such a way that it is very similar to the student body as a whole. How that can be done is outlined below. It is important to stress, however, that such a sample-even one that can be demonstrated to be very similar to the entire student body on characteristics like gender, race and ethnicity, year in college, and so forth-will not be truly representative of that student body. Only a random sample can be described that way. These alternative methods can be made acceptable, but they remain second- best options and should be recognized as such. Administering the Survey During Course Registration. One alternative method for administering the survey is to integrate the data collection process into other campus routines, such as course registration. This will work only if all (or nearly all) students are required to report to a central location during some part of the registration procedure. Registration officials will need to be contacted in advance to work out arrangements. Ideally, approaching students about participating will be built in as a step in the registration process. Space will be needed for up to 10 students at a time to complete the survey in relative privacy. Staff provided by the survey team should be available to distribute and collect the survey forms and to answer questions. To make sure that students know that the information they provide will remain anonymous, they should receive a memorandum that explains the purpose of the survey and how the completed questionnaires will be collected and stored. Each student should be provided with an unmarked envelope in which to put the completed survey. With a small enough student body, it might be possible to have every student complete the survey. Institutions with larger enrollments will need to select every Xth student in line. For example, with a student enrollment of 20,000, every 25th student could be approached to generate a sample size of 800 students. Selecting every Xth student could result in a random sample, if all students were required to come to a central location to register, and if the selection procedure were followed perfectly. That is unlikely to happen, however. If the research staff are able to cover registration only during certain time periods, or if a heavy flow of students makes it physically impossible to ensure that every Xth one is selected, a nonrandom sample will result. Participation should not be required. Even so, high rates of participation can be built by making an effective personal appeal, giving a careful explanation of the procedures, and providing extra incentives, such as snacks, coupons, or eligibility for prize drawings. Administering the Survey During Classes. A second alternative method for administering the survey is to distribute it during classes. This method is inexpensive. The only significant costs are for the test administrators, who might instead be recruited as volunteers. This method is also quick. Surveys can usually be administered within a week. Again, selecting a convenience sample of this sort is not recommended as a first choice, because as usually practiced, it does not result in a sample that is truly representative of the student body as a whole. Even so, administering the survey during classes may be the only method that colleges and universities without adequate resources to conduct a random sample survey will be able to use. It should be noted that there are conditions under which this procedure can generate a true random sample. Specifically, there must be a class (or a type of class) that each and every student is required to take at the same time, such as freshman English or physical education. Ideally, each class section should have approximately the same number of students. Weighting procedures can be applied to correct for having class sections of different sizes; consult a statistician for additional information. In administering the survey, the researchers draw a random sample of class sections and then ask every student in those sections to complete the instrument. By this method, each student has an equal probability of being selected into the sample, which is the defining characteristic of a random sample. The statistical method for calculating "confidence intervals" is different for this type of sample compared with a simple random draw of students; consult a statistician for additional information. At most colleges, however, these conditions cannot be met, and therefore a true random sample of students cannot be drawn using this procedure. At best, even if the researchers take great care, the sample will be only an approximation of a random sample. Does this matter? Yes. The problem is that selecting a nonrandom sample of students can lead to inaccurate results. The same type of inaccuracy can creep in if classes are selected by convenience rather than at random. If only English or mathematics classes were chosen, the source of the expected bias would be obvious. If only small classes were chosen, there would be a different type of bias. The point is that, no matter how the classes are chosen, if they are chosen non-randomly, some kind of bias can be expected. The amount of bias can be controlled somewhat. By selecting the right classes, researchers can draw a student sample that matches the total student body in several important respects (e.g., gender, age, race and ethnicity, year in college). Doing this is a major advantage. Even then, however, researchers must accept the fact that this student sample would not be truly representative of the student body as a whole. This is not said to discourage campus officials from doing a survey, but to remind them of the real limitations presented by this survey method. __________________ Sidebar: Going Beyond Student Surveys As a supplement to self-report surveys, structured field observations, with locations and times of days selected at random, can be used to assess student intoxication by measuring blood alcohol concentration (BAC) with a passive breathalyzer. Archival records are another useful supplement: (1) the number of students seen in local emergency rooms or the student health center for alcohol-related injuries or illnesses, (2) the number of students arrested for AOD-related infractions, (3) building and equipment damage reports, and (4) the number of residence hall complaints due to AOD-related behavior. How well these records are kept might change over time, which could make these data difficult to interpret. In fact, on most campuses, the evaluation team will need to work with the various campus departments (e.g., campus police, student health services) and community agencies (e.g., local police, hospital ER) to develop forms and record-keeping procedures to improve the quality of information received. __________________ Unfortunately, not every institution of higher education will have the technical capability to identify a set of classes that will result in a good sample. The college's computer records must include key demographic information about its students, and there must be accurate, up-to-date records of which classes each student is taking. Moreover, all of this information must be stored in a database that can be easily manipulated to identify the best classes to select. There also is the difficulty of getting faculty to grant permission for the survey to be conducted in their classes. Many instructors will cooperate, but some may be reluctant to set aside the time. Researchers have found it helpful to solicit senior administrator support, explaining the importance of surveying all randomly selected classrooms and the benefits of collecting the data. Faculty may also respond to an appeal that acknowledges the students' loss of instruction time but also underscores the benefits to the entire campus community from the knowledge acquired through the survey. Even so, researchers may not get access to all of the classes, which must be taken into account when judging whether the resulting student sample is representative. With classroom administration, members of the research team, not class instructors, should administer the survey. This should be done at the beginning of the class period to make sure there is enough time. As with other administration procedures, due care must be taken to ensure that students know that their participation is voluntary and that their responses will remain anonymous. This message can be underscored by not walking around the classroom as the surveys are being completed, reminding students to respect one another's privacy, and collecting the completed surveys in a single box or envelope. Analyzing the Data Most campus administrators will need to do only two types of data analysis to get a good picture of student drinking at their institution: descriptive statistics and tests of association. Administrators interested in more sophisticated data analyses (e.g., significance tests, multiple regression) will need to consult with statisticians-possibly those on their own faculty. Descriptive statistics are used to summarize the students' survey responses. Such statistics are used to answer questions such as the following: (1) What percentage of students at the college can be classified as heavy drinkers? (2) On average, how many drinks per week do students consume? Counting how many students fall into a category produces a statistic known as the frequency. A frequency table itemizes the number of students in each of several categories. The percentage of students in each category can also be calculated, based on either the total number of responses or the total number of valid responses (that is, with respondents with missing information excluded). Frequency data are often reported using pie diagrams or bar graphs. The results for question 1 above could be presented this way. __________________ Sidebar: Rates of heavy, episodic drinking among college students vary greatly across U.S. colleges and universities. A seminal survey of 140 four-year colleges found that the percentage of students who drank heavily ranged from 1 percent at the college with the lowest incidence to 70 percent at the college with the highest.21 One reason for this wide variation is that the student bodies at these colleges are different, and certain types of students are more likely to be heavy drinkers than others.22 For example, white students are several times more likely to be heavy drinkers than students of color. Thus, all other things being equal, colleges with a higher percentage of white students would be expected to have a greater number of heavy drinkers. Other critical factors include gender, age, marital status, residential status, and membership in fraternities or sororities. The implication is clear: An individual college should not compare its heavy drinking rates with overall national averages without taking into account the nature of the institution and its student body. __________________ A useful way to summarize a data set involving rating scales or other numerical scores is to report the average or typical measurement, what statisticians call the data set's central tendency. There are three measures of central tendency: mode, median, and mean. The mode is the score that occurs most frequently. Note that a data set can have more than one mode if the maximum frequency is shared by two or more scores. The median is the score that separates the upper half of the scores from the lower half. That is, 50 percent of the scores are larger than the median, and 50 percent are smaller. The mean is what most people think of as the "average." To calculate the mean, add up all the scores and divide that sum by the total number of scores. Tests of association such as the X2 (chi-square) test or Pearson's product - moment correlation coefficient are used to assess the existence of a relationship between two variables. For example, administrators might want to know whether there is a significant relationship between how many drinks students usually consume when they drink and the amount they drank the last time they rode with a designated driver. Tests of association also can be used to examine whether two or more groups have truly different frequency distributions for some measurement. For example, campus administrators might ask whether a greater number of fraternity members are frequent heavy drinkers than nonmembers. Reporting the Results Student surveys are used to determine the scope of AOD use and its consequences, ascertain the individual and environmental factors that contribute to the problem, and assess the reach and effectiveness of prevention programs and policies. Baseline surveys are a critical part of a problem analysis, the first step in a systematic strategic planning process.23 Later surveys are used to monitor progress in meeting program objectives and reducing AOD-related problems.24 Many college administrators are reluctant to report publicly the results of their institution's student surveys, even when there has been progress in reducing AOD-related problems. As noted before, collecting and using survey data to develop, monitor, and refine prevention programming is consistent with best practice in the field. Given that essential purpose, it is counterproductive to hold back on publicly reporting any survey results or other research findings. The major concern, of course, is that the survey findings will bring negative attention to the institution. It must be remembered, however, that the public already knows about college students' involvement in substance abuse, especially illegal and excessive alcohol consumption. In short, this is not news. Also, administrators should consider that in most cases the data will show that the problem is less severe than the public imagines, based on the news reports, movies, and television shows they see. Admitting that a minority of students attending the college are part of a national problem is no disgrace. Rather, reporting and discussing these data are the first steps in bringing focused attention to the problem and building support for evidence-based programs and policies. Even if later surveys show that progress is slow or remains elusive, the need for public disclosure remains just as compelling. Confronting reality, not hiding from it, is the key to galvanizing a stronger commitment to address the problem. When institutions are making progress in combating substance use, a failure to report survey findings is not just counter- productive, but self-defeating. A college that is widely known as a "party school" will continue to attract high- risk students who are looking for that kind of experience, and misperceptions of actual student drinking norms will continue to perpetuate the problem. If the college's reputation is to catch up with reality, then good information about its efforts to confront the problem and any encouraging survey information should be proclaimed far and wide. Conclusion Ideally, college administrators will commit themselves to a long-term program of research so that they can monitor changes in student alcohol use and other campus conditions. This is essential for assessing the effect of an institution's prevention programs and policies. The student survey methods outlined in this guide should be a key component of that research effort. This guide has shown that the best survey method involves selecting a random sample of students and then sending them a survey by mail or online, with a strong follow-up effort made to generate a high response rate. Due to limited resources, not all institutions are in a position to administer their surveys in this way, and they must consider second-best alternative methods, such as administering surveys to classes. As discussed, these methods can be conducted in a way to minimize their limitations and produce usable, if not ideal, results. Given the need for reliable and accurate information, and given the clear superiority of studying a true random sample of students, college officials should consider increasing their research budget to make it possible to use the preferred survey method. The relatively modest resources that are needed are not excessive, especially considering the improved quality of information that would result. Ultimately, good decision-making must rest on a foundation of accurate information. William DeJong, Ph.D., is a professor of social and behavioral sciences at the Boston University School of Public Health and a senior adviser to the Higher Education Center for Alcohol and Other Drug Abuse and Violence Prevention. References 1. DeRicco, B. Complying With the Drug-Free Schools and Campuses Regulations [EDGAR Part 86]: A Guide for University and College Administrators (Washington, D.C.: U.S. Department of Education, Higher Education Center for Alcohol and Other Drug Abuse and Violence Prevention, 2006). 2. Lake, P. "Law as Prevention." Prevention File: Alcohol, Tobacco and Other Drugs 18(2): 5-7, 2003. 3. Task Force of the National Advisory Council on Alcohol Abuse and Alcoholism, National Institute on Alcohol Abuse and Alcoholism. A Call to Action: Changing the Culture of Drinking at U.S. Colleges (Washington, D.C.: National Institutes of Health, 2002). 4. U.S. Department of Transportation, National Highway Traffic Safety Administration. "Computing a BAC Estimate." Available at http://www.nebraskaprevlink.ne.gov/therightstuff/community/LB166/BAC.html. (Accessed on June 16, 2007.) 5. DeJong, W.; Schneider, S. K.; Towvim, L. G.; Murphy, M. J.; Doerr, E. E.; Simonsen, N. R.; Mason, K. E.; and Scribner, R. A. "A Multisite Randomized Trial of Social Norms Marketing Campaigns to Reduce College Student Drinking." Journal of Studies on Alcohol 67: 868-879, 2006. 6. Aday, L. A. Designing and Conducting Health Surveys (2nd ed.) (San Francisco: Jossey-Bass, 1996); Jackson, C. J., and Furnham, A. Designing and Analyzing Questionnaires and Surveys: A Manual for Health Professionals and Administrators (Philadelphia, Pa.: Whurr Publishers Ltd., 2000). 7. DeJong et al., "A Multisite Randomized Trial." 8. Dillman, D. A.; Eltinge, J. L.; Groves, R. M.; and Little, R. J. A. "Survey Nonresponse in Design, Data Collection, and Analysis." In Groves, R. M.; Dillman, D. A.; Eltinge, J. S.; and Little, R. J. A. (ed.), Survey Nonresponse (New York, N.Y.: John Wiley & Sons, 2002), pp. 3-26. 9. Pealer, L.; Weiler, R.; Pigg, R.; Miller, D.; and Dorman, S. "The Feasibility of a Web-Based Surveillance System to Collect Health Risk Behavior Data from College Students." Health Education & Behavior 28: 547-559, 2001. 10. Miller, E. T.; Neal, D. J.; Roberts, L. J.; Baer, J. S.; Cressler, S. O.; Metrik, J.; and Marlatt, G. A. "Test-Retest Reliability of Alcohol Measures: Is There a Difference Between Internet-Based Assessment and Traditional Methods?" Psychology of Addictive Behaviors 16: 56-63, 2002. 11. Jones, S. The Internet Goes to College: How Students Are Living in the Future with Today's Technology (Philadelphia, Pa.: Pew Charitable Trusts, 2002). Available at http://www.pewinternet.org/report_display.asp?r=71. (Accessed on June 17, 2007.) 12. Green, K. C. Campus Computing 2000: The 11th National Survey of Computing and Information Technology in American Higher Education (Encino, Calif.: Campus Computing, 2001). 13. DeJong et al., "A Multisite Randomized Trial"; Wechsler, H.; Lee, J. E.; Kuo, M.; Seibring, M.; Nelson, T. F.; and Lee, H. "Trends in College Binge Drinking During a Period of Increased Prevention Efforts: Findings from 4 Harvard School of Public Health College Alcohol Study Surveys: 1993-2001. Journal of American College Health 50: 203-217, 2002. 14. McCabe, S. E.; Boy, C. J.; Couper, M.; and Crawford, S. Web-Based Approaches for Evaluating Alcohol and Other Drug Prevention Efforts. Paper presented at the U.S. Department of Education's 15th Annual National Meeting on Alcohol, Other Drug, and Violence Prevention in Higher Education, Washington, D.C., 2001. 15. Dowdall, G. W., and Wechsler, H. "Studying College Alcohol Use: Widening the Lens, Sharpening the Focus." Journal of Studies on Alcohol, supplement no. 14: 14-22, 2002. 16. Edwards, P.; Roberts, I.; Clarke, M.; DiGuiseppi, C.; Pratap, S.; Wentz. R.; and Kwan, I. "Increasing Response Rates to Postal Questionnaires: Systematic Review." British Medical Journal 324: 1183-1191, 2002. 17. Ibid. 18. DeJong et al., "A Multisite Randomized Trial." 19. Ibid. 20. Schaefer, D., and Dillman, D. "Development of a Standard E-mail Methodology: Results of an Experiment." Public Opinion Quarterly 62: 378-397, 1998. 21. Wechsler, H.; Davenport, A.; Dowdall, G. W.; Moeykens, B.; and Castillo, S. "Health and Behavioral Consequences of Binge Drinking in College: A National Survey of Students at 140 Campuses." Journal of the American Medical Association 272: 1672-1677, 1994. 22. Wechsler et al., "Trends in College Binge Drinking." 23. Langford, L., and DeJong, W. Strategic Planning for Prevention Professionals on Campus (Washington, D.C.: U.S. Department of Education, Higher Education Center for Alcohol and Other Drug Abuse and Violence Prevention, 2008). 24. DeJong, W., and Langford, L. M. Evaluating Environmental Management Approaches to Alcohol and Other Drug Abuse Prevention (Washington, D.C.: U.S. Department of Education, Higher Education Center for Alcohol and Other Drug Abuse and Violence Prevention, 2006). Sources of Information Office of Safe and Drug-Free Schools (OSDFS) U.S. Department of Education http://www.ed.gov/osdfs; 202-245-7896 OSDFS supports efforts to create safe schools, respond to crises, prevent alcohol and other drug abuse, ensure the health and well-being of students, and teach students good character and citizenship. The agency provides financial assistance for drug abuse and violence prevention programs and activities that promote the health and well-being of students in elementary and secondary schools and institutions of higher education. The U.S. Department of Education's Higher Education Center for Alcohol and Other Drug Abuse and Violence Prevention http://www.higheredcenter.org; 1-800-676-1730; TDD Relay-friendly, Dial 711 The Higher Education Center considers strategic planning and evaluation to be an important component of a comprehensive prevention approach. The Higher Education Center has several publications and other materials to help campus administrators develop and evaluate prevention programs. These materials (examples listed below) can be accessed for free from the Publications and Evaluation sections of the Center's Web site. The Network Addressing Collegiate Alcohol and Other Drug Issues http://www.thenetwork.ws; see Web site for telephone contacts by region The Network Addressing Collegiate Alcohol and Other Drug Issues (Network) is a national consortium of colleges and universities formed to promote healthy campus environments by addressing issues related to alcohol and other drugs. Developed in 1987 by the U.S. Department of Education, the Network comprises member institutions that voluntarily agree to work toward a set of standards aimed at reducing AOD problems at colleges and universities. Publications College Alcohol Risk Assessment Guide: Environmental Approaches to Prevention by B. E. Ryan; T. Colthurst; and L. Segars This guide outlines methods for identifying and analyzing factors in the campus and community environment that contribute to alcohol-related problems (104 pp., 1997). How to Select a Program Evaluator by L. Langford and W. DeJong This document describes the skills, expertise, and experience to look for when seeking an evaluator; questions to ask when assessing an evaluator's past work; and guidance on how to network to find the right person and forge an effective working relationship (3 pp., 2001). Evaluating Environmental Management Approaches to Alcohol and Other Drug Abuse Prevention by W. DeJong and L. M. Langford This document outlines the basic steps for evaluating a program for alcohol and other drug abuse prevention that features environmental change efforts (6 pp., 2006). Acknowledgments We wish to thank the individuals listed below for reviewing draft manuscripts of this publication. We appreciate the comments they provided to help us assure that this publication has a solid scientific foundation and contains clear messages. To the extent that we achieved that goal, the credit is theirs. To the extent we did not, the fault is ours. * Peggy Glider, University of Arizona, Tucson, Ariz. * H. Wesley Perkins, Hobart and William Smith Colleges, Geneva, N.Y. * Robert F. Saltz, Prevention Research Center, Berkeley, Calif. This version of Methods for Assessing College Student Use of Alcohol and Other Drugs is a revision by William DeJong of the earlier version that was prepared in 1995 by W. DeJong and H. Wechsler. Titles in the Prevention 101 Series The Approach and Framework 1. Environmental Management: A Comprehensive Strategy for Reducing Alcohol and Other Drug Use on College Campuses 2. Environmental Management: An Approach to Alcohol and Other Drug Prevention 3. Experiences in Effective Prevention: The U.S. Department of Education's Alcohol and Other Drug Prevention Models on College Campuses Grants 4. Preventing Violence and Promoting Safety in Higher Education Settings: Overview of a Comprehensive Approach The Building Blocks 1. Getting Started on Campus: Tips for New Prevention Coordinators 2. Strategic Planning for Prevention Professionals on Campus 3. Problem Analysis: The First Step in Prevention Planning [In review.] 4. Setting Goals and Choosing Evidence-based Strategies [In review.] 5. College Alcohol Risk Assessment Guide: Environmental Approaches to Prevention (CARA) 6. Methods for Assessing Student Use of Alcohol and Other Drugs 7. Evaluating Environmental Management Approaches to Alcohol and Other Drug Abuse Prevention 8. Alcohol and Other Drug Policies for Colleges and Universities: A Guide for Administrators [In review.] Our Mission The mission of the U.S. Department of Education's Higher Education Center for Alcohol and Other Drug Abuse and Violence Prevention is to assist institutions of higher education in developing, implementing, and evaluating alcohol, other drug, and violence prevention policies and programs that will foster students' academic and social development and promote campus and community safety. How We Can Help The U.S. Department of Education's Higher Education Center offers an integrated array of services to help people at colleges and universities adopt effective prevention strategies: * Resources, referrals, and consultations * Training and professional development activities * Publication and dissemination of prevention materials * Assessment, evaluation, and analysis activities * Web site featuring online resources, news, and information * Support for the Network Addressing Collegiate Alcohol and Other Drug Issues Get in Touch Additional information can be obtained by contacting: The Higher Education Center for Alcohol and Other Drug Abuse and Violence Prevention Education Development Center, Inc. 55 Chapel Street Newton, MA 02458-1060 Web site: http://www.higheredcenter.org Phone: 1-800-676-1730; TDD Relay-friendly, Dial 711 E-mail: HigherEdCtr@edc.org Funded by the U.S. Department of Education