PREVENTION UPDATES The Higher Education Center for Alcohol and Other Drug Abuse and Violence Prevention February 2006 Evaluating Environmental Management Approaches to Alcohol and Other Drug Abuse Prevention by William DeJong, Ph.D. and Linda M. Langford, Sc.D. Recent years have seen an upsurge in prevention work focused on changing the campus and community environments in which college students make decisions about alcohol and other drug (AOD) use. This approach, called environmental management, is based on three fundamental premises: 1. Substance use problems are aggravated by a physical, social, economic, and legal environment that increases both the availability and appeal of alcohol and other drugs.1 2. The most cost-effective means of reducing the number of AOD problems that a community experiences is to change that environment, thereby increasing motivation to avoid illegal or excessive substance use and its negative consequences.2 3. Successful environmental management requires a coalition of campus and community officials to develop and execute a strategic plan that features science-based programs and policies.3 Environmental management strategies can be organized into five major categories: (1) offering alcohol-free social, extracurricular, and public service options; (2) creating a health-promoting normative environment; (3) limiting alcohol availability; (4) restricting the marketing and promotion of alcohol; and (5) creating and increasing enforcement of policies and laws. All five strategies involve a wide range of possible program and policy options (see sidebar "Five Strategic Objectives Focused on Environmental Change").4 A comprehensive program will include multiple interventions targeted to local problems. ____________________________ Sidebar: Five Strategic Objectives Focused on Environmental Change Substance use problems are driven by five environmental factors that increase both the availability and the appeal of alcohol and other drugs, each of which can be addressed by the following set of environmental management strategies: 1. Alcohol-Free Options: Many students, especially at residential colleges, have few adult responsibilities, a great deal of unstructured free time, and too few social and recreational options. The strategic objective: offer and promote social, recreational, extracurricular, and public service options that do not include alcohol and other drugs. 2. Normative Environment: Many people accept drinking and other drug use as a ?normal? part of the college experience. The strategic objective: create a social, academic, and residential environment that supports health-promoting norms. 3. Alcohol Availability: Alcohol is abundantly available to students and is inexpensive. The strategic objective: limit alcohol availability both on and off campus. 4. Alcohol Marketing and Promotion: Local bars, restaurants, and liquor stores use aggressive promotions to target underage and other college drinkers. The strategic objective: restrict marketing and promotion of alcoholic beverages both on and off campus. 5. Policy Development and Enforcement: Campus policies and local, state, and federal laws are not enforced consistently. The strategic objective: develop and enforce campus AOD policies and local, state, and federal laws. ______________________________ Health educators and counselors who run traditional education and treatment programs work to persuade individual students to lower their substance use, but such change is much more difficult when the campus and community environments continue to facilitate or even encourage that high-risk behavior. Successful prevention requires a wide-ranging approach that has environmental management as its foundation. Conscientious academic administrators who apply environmental prevention approaches will want to know if their efforts are effectively reducing alcohol and other drug problems (see sidebar "Principles of Effectiveness for Prevention Programs"). This requires systematic evaluation-a process for collecting, analyzing, and reporting information that can answer that question. This publication is designed to provide interested college and university administrators with a brief orientation to the evaluation of prevention programs that are based on the environmental management approach. In practice, campus officials who have applied an environmental management approach to AOD abuse prevention have found it useful to view program development and evaluation as an iterative process, with evaluation findings helping to inform program modifications. The evaluation process has five steps: (1) describing the intervention; (2) identifying process measures; (3) identifying outcome measures; (4) selecting a research design; and (5) utilizing the results. _______________________ Sidebar: Principles of Effectiveness for Prevention Programs The U.S. Department of Education's Office of Safe and Drug-Free Schools promotes principles of effectiveness for prevention programs, as codified in the No Child Left Behind Act of 2001. A subset of the principles of effectiveness that are most applicable to institutions of higher education can be summed up as follows: * Design programs based on a thorough needs assessment of objective data. * Establish a set of measurable goals and objectives linked to identified needs. * Implement prevention activities that research or evaluation have shown to be effective in preventing high-risk drinking or violent behavior. * Use evaluation results to refine, improve, and strengthen the program and refine goals and objectives as appropriate. ________________________ Describing the Intervention A strategic plan based on the environmental management approach usually entails several strategies that may evolve and change over time. Therefore, an essential first step in the evaluation is to create a summary of the prevention effort by developing a logic model or evaluation map. The purpose of the logic model is to list the program's key components and then outline how each one is supposed to work-that is, to show the chain of events that is anticipated, moving from specific strategies and activities to intermediate and long-term outcomes. Figure 1 is a logic model for a responsible beverage service (RBS) program, an environmental strategy designed primarily to prevent excessive consumption of, and reduce minors' access to, alcohol. The logic model serves as the basis for the evaluation plan, noting both the interim steps and outcomes that can be measured and assessed. If an evaluation were to show that a program's goals and objectives were not achieved, staff could ascertain where in the logic model's expected sequence an activity did not work as planned (e.g., high turnover resulting in some staff not being trained in RBS). If the expected outcomes were obtained, having an evaluation that examined each step in the logic model would help to establish that the activity contributed directly to achieving the program's goals and objectives. It is important that an evaluator be part of the planning process at this early stage.5 The evaluator may help to ensure that: (1) the needs assessment is well designed and can provide baseline (i.e., pre-intervention) data for the evaluation; (2) the intervention plan features policies and programs with demonstrated effectiveness or a solid foundation in behavior change theory; (3) the program's goals and objectives are precisely stated so that measurable outcomes can be specified; (4) each program component and policy can be linked logically to specific objectives; and (5) adequate resources are in place to ensure full and adequate implementation of the program plan. ___________________________ Figure 1. Responsible beverage service (RBS) logic model Program Resources Campus and community coalition to provide management oversight Local government: * Procedure for licensing RBS training programs * Procedure for certifying RBS graduates * RBS recertification requirement RBS protocols for bar and restaurant staff and management Certified RBS training program for staff and managers * Factual knowledge * Supportive attitudes * Skills Agreement with taxicab company to provide alternative transportation Graduated administrative penalties for licensees for noncompliance Program Activities Bar and restaurant staff and managers receive RBS training. Campus officials conduct media campaign to inform students about RBS procedures. State or local law enforcement officials conduct: * Surveillance of RBS training programs * Random compliance checks Intermediate Outcomes Bar and restaurant staff: * Refuse entrance to minors or intoxicated patrons * Refuse service to patrons who could become intoxicated * Arrange alternative transportation for patrons who should not drive * Provide free food and soft drinks for designated drivers Managers: * Issue reminders about RBS practices * Monitor and correct staff performance * Provide refresher training as needed * Arrange for new employees to receive RBS training Long-Term Outcomes Underage students will purchase less alcohol at local bars and restaurants. Students will drink less alcohol at local bars and restaurants. Students will drive less often when intoxicated. Students will ride less often with an intoxicated driver. Health and Quality of Life Students will suffer fewer negative health consequences due to their own or other students? drinking. Local residents will register fewer noise and nuisance com-plaints due to student drinking. Source: This logic model was developed by W. DeJong and L. Langford, Higher Education Center for Alcohol and Other Drug Abuse and Violence Prevention, 2005. _________________________________ Identifying Process Measures The second step in the evaluation process is to develop process measures for documenting the nature, extent, and quality of program implementation. Stated simply, process measures are used to assess whether the prevention effort is being implemented as designed. This information is critical. Once the intervention is under way, process data may be used to monitor progress and determine whether corrective steps are needed. If a particular intervention were to fail, having the process data would make it possible for evaluators to determine whether that failure might have been due to inadequate implementation. One group of process measures looks at utilized resources-that is, the staff, volunteers, and funds used to operate the program, which might differ from what was originally planned. Typical measures include a count of full-time equivalents (FTEs) and expenditures. Another group of process measures looks at activity levels and the numbers of people or groups being served by the program. For an RBS program, measures include the number of bars and restaurants signing up for the RBS program, the number of staff and managers receiving training, and the number of random compliance checks and other enforcement activities. A key process in environmental management is the formation and operation of a campus and community coalition.6 Certain process indicators are linked to the activities of the coalition itself-establishing a core leadership team and a subcommittee structure; developing rules of governance; recruiting membership; conducting a needs assessment, including a scan of the campus and community environment; identifying science-based program and policy options; outlining a strategic plan, including goals, objectives, environmental management strategies and tactics; and implementing that plan. Also critical are process measures of coalition function-member commitment to the coalition's goals; degree and quality of participation; perceived support from high-level campus and community officials; and degree of satisfaction with the coalition's leadership and structure, the conduct of meetings and other activities, communication and outreach, and methods for decisionmaking and conflict resolution. The most important question, however, is whether the structure and function of the coalition helps campus and community officials to achieve the environmental changes specified in the strategic plan. Identifying Outcome Measures Outcome measures to assess a program's success or failure also need to be identified. There are two types of outcomes to consider: changes in behavior, and changes in structure or functioning of the environment. Behavioral Outcomes When specifying behavioral goals and objectives, it is important to differentiate between intermediate and long-term outcomes (see fig. 1). The prevention program's logic model will specify a set of intermediate outcomes that are necessary for achieving the desired long-term outcomes. For an RBS program, for example, intermediate outcomes include several staff behaviors outlined in the RBS protocol-refusing entrance to minors or intoxicated patrons, refusing service to patrons who could become intoxicated, arranging alternative transportation for patrons who should not drive, and providing free food and soft drinks for designated drivers. Looking at long-term outcomes, evaluation plans usually include measures of student behavior, because the ultimate goal of any prevention program is to reduce AOD problems among students, including illegal alcohol and other drug use. A typical RBS program assesses the purchase and consumption of alcohol and the use of alternative rides or designated drivers to avoid driving after drinking. The best method for assessing student behavior is a survey administered to a randomly drawn sample of students. Self-reports, when provided under conditions of confidentiality, are known to produce population-level estimates that are generally both valid and reliable.7 Pencil-and-paper surveys work, but Web-based surveys, with participants recruited by e-mail, are increasingly the method of choice due to lower costs and the ease and speed of administration. These two methods generally produce comparable response rates.8 Existing surveys may be examined for questions to include.9 To create the study sample, most college registrars are able to generate a randomly chosen list of enrolled undergraduates. In general, administrators should send surveys to a sample of between 400 and 1,000 students. If the school's budget permits it, a sample of 1,000 is far superior. Even under the severest budget constraints, the sample should not be any smaller than 200. 10 Essential to a successful survey is a high response rate. While researchers generally want to achieve response rates of 70 percent or higher, recent national surveys have typically reached rates of between 50 and 60 percent.11 Factors that may increase the response rate include procedures to protect student confidentiality, pre-survey messages notifying students about the survey, sending multiple reminders, and offering financial or other incentives for completing and returning the survey. Response rates also tend to be higher with shorter surveys.12 Student behavior also may be assessed by structured field observations of student behavior. Measures might include numbers of student patrons at alcohol- free venues or numbers of students who are carrying drinks in public. Structured field observations also may be used to assess student intoxication by measuring blood alcohol concentration (BAC) with a passive breathalyzer. Unobtrusive measures also are possible-for example, the volume of alcohol-related litter (cans, bottles, drinking cups). Outcome measures also may include AOD consequences. Surveys may ask students directly about consequences resulting either from their own or other students' AOD use. Information from various archival records may be useful, too-for example, the number of students seen in local emergency rooms or the student health center for alcohol-related injuries or illnesses, number of students arrested for AOD-related infractions, building and equipment damage reports, and number of residence hall complaints due to AOD-related behavior. How well these records are kept might change over time, which could make these data difficult to interpret. Environmental Outcomes An evaluation of programs focused on environmental change should not be restricted to measures of student behavior and AOD-related consequences. Seeing meaningful change at this level might require several years, and there are many other factors, beyond the control of campus and community officials, that might influence individual outcomes (e.g., changes in alcohol excise taxes, new alcohol products, greater availability of low-priced illicit drugs). Also important are measures of the environment itself.13 Measuring the environment as part of needs assessment and evaluation has two key benefits- identifying and raising awareness of environmental factors that contribute to AOD problems, and documenting intermediate outcomes of environmental change efforts that set the stage for changes in individual student behavior. There are two primary categories for environmental measurement: (1) self- reports; and (2) archival records and program documents. Self-reports may come from surveys, interviews, or focus groups with students, faculty, community residents, and others. Written records also are an important data source. On most campuses, the evaluation team will need to work with the various campus departments (e.g., campus police, student health services) and community agencies (e.g., local police, hospital emergency room) to develop forms and recordkeeping procedures to improve the quality of information received. Measures should be chosen that correspond to the particular interventions being employed and the specific environmental changes that are outlined in the logic model. Such measures, organized by environmental management strategy, include the following: Alcohol-Free Options Student Self-Report: (1) percentage reporting attendance at substance-free social events; (2) number of hours spent in volunteer public service. Archival Records: (1) number of student organizations given funds to support substance-free events; (2) number of hits on a college-sponsored Web site listing recreational alternatives in the community. Normative Environment Student Self-Report: (1) reported exposure to messages about true drinking norms on campus, designed to correct existing misperceptions; (2) reports of mixed messages about AOD use from faculty, resident assistants, and other staff. Archival Records: (1) number of beds and the occupancy rate for substance-free housing on campus; (2) percentage of academic courses holding Friday classes. Alcohol Availability Student Self-Report: (1) perceived ease of acquiring alcohol from social and commercial sources; (2) average price paid for standard alcohol products (e.g., six-pack of beer). Archival Records: (1) number of liquor licenses within one mile of campus; (2) number of bars and restaurants promoting "happy hours" in campus newspaper advertisements. Alcohol Marketing and Promotion Student Self-Report: (1) reported exposure to party announcements that mention drinking; (2) reported exposure to bar advertisements in campus and local newspapers. Archival Records: (1) number of alcohol advertisements per week appearing in the student newspaper; (2) number of alcohol industry-sponsored events at local bars and restaurants. Policy Development and Enforcement Student Self-Report: (1) perceived risk of arrest for driving under The influence (DUI); (2) reported exposure to information about the state's "zero tolerance" law for underage students. Archival Records: (1) number of special operations to enforce the minimum-age drinking law; (2) number of students cited for AOD-related conduct violations. In some cases, systematic field observations also may be used to assess environmental change. Indicators include the number of kiosk messages that promote high-risk drinking, the number of price discounts being offered by local liquor stores, and the availability of alcohol- and other drug-related paraphernalia in campus and community stores. Selecting a Research Design A research design is the basic structure of the evaluation, which outlines when and where data will be collected to study the effect of the environmental management program. Typically, an evaluation plan will call for data collection both before (pretest, or baseline) and after (posttest) the prevention program is launched. Positive changes in the intermediate and long-term outcomes might be taken as a sign that the program has been successful. The problem with this simple one group pretest-posttest design is that there are many other possible explanations for the observed changes. For example, there might have been other changes occurring at the community or state level that had no direct connection with the intervention yet had the potential to influence student behavior. Or perhaps campus police, as they became more experienced with a new recordkeeping form, were more likely to note whether a reported incident involved student alcohol use. The purpose of more advanced research designs-involving data collection at nonintervention sites-is to increase confidence that any outcome changes that coincide with the program can actually be attributed to it. The most common design is a nonequivalent comparison group design, which involves pretest and posttest data collection at one or more sites with the program ("treatment group" or "experimental group") and at one or more sites without the program ("comparison group" or "control group"). Building in this comparison makes it possible to rule out many competing explanations for any pretest-to-posttest changes in a given treatment group. Using an advanced design is less important if the environmental management strategies being used are already known to be effective-if not with college students then with the general population-for example, responsible beverage service, higher alcohol prices, increased enforcement of the minimum-age drinking law, or increased DUI enforcement.14 If that is not the case, then whether the research design can be expanded to include comparison sites will depend on funding availability and the cooperation of officials from a comparable campus community, who need to agree to continue holding back on implementing environmental management strategies. This is not always feasible. In most cases, a more reasonable alternative might be to examine how any observed changes in long-term outcomes compare with those being seen statewide or nationally at other campus communities. At the very least, campus administrators need to know whether their key indicators are moving in the right direction. Utilizing the Results The main reason for conducting an evaluation is to find out if a prevention program is working as intended. Process measures are used to document how the program was implemented, while outcome measures are used to establish the program's effect on key environmental factors that affect student behavior, actual levels of students' substance use, and the consequences of that use. The combined use of process and outcome measures can help administrators develop plans for strengthening or improving the program. If an environmental approach appears to be successful, then consider broadening its scope or investing additional resources. If a particular approach appears to have failed, then diagnose what went wrong. Was the program implemented as planned? If not, can that be corrected? Might the approach have succeeded if even more resources were put into it? Or does it need to be abandoned in favor of a new idea? Stated simply, evaluation is a management tool. Understanding the findings, and then using them to make program changes, requires the same deliberateness and care that originally went into developing, implementing, and evaluating the prevention program. William DeJong, Ph.D., is a professor of social and behavioral sciences at the Boston University School of Public Health and a senior Center adviser to the Higher Education Center for Alcohol and Other Drug Abuse and Violence Prevention. Linda M. Langford, Sc.D., is associate director for violence prevention initiatives at the Higher Education Center for Alcohol and Other Drug Abuse and Violence Prevention. References 1. DeJong, W.; Vince-Whitman, C.; Colthurst, T.; Cretella, M.; Gilbreath, M.; Rosati, M.; and Zweig, K. Environmental Management: A Comprehensive Strategy for Reducing Alcohol and Other Drug Use on College Campuses (Washington, D.C.: U.S. Department of Education, Higher Education Center for Alcohol and Other Drug Prevention, 1998); Holder, H. Alcohol and the Community: A Systems Approach to Prevention (Cambridge, U.K.: Cambridge University Press, 1999). 2. Ibid. 3. Hingson, R. W., and Howland, J. "Comprehensive Community Interventions to Promote Health: Implications for College-Age Drinking Problems." Journal of Studies on Alcohol, supplement no. 14: 226-240, 2002. 4. Zimmerman, R., and DeJong, W. Safe Lanes on Campus: A Guide for Preventing Impaired Driving and Underage Drinking (Washington, D.C.: U.S. Department of Education, Higher Education Center for Alcohol and Other Drug Prevention, 2003). 5. Langford, L. M., and DeJong, W. Prevention Update: How to Select a Program Evaluator (Washington, D.C.: U.S. Department of Education, Higher Education Center for Alcohol and Other Drug Prevention, 2001). 6. Hingson and Howland, "Comprehensive Community Interventions." 7. Dowdall, G. W., and Wechsler, H. "Studying College Alcohol Use: Widening the Lens, Sharpening the Focus." Journal of Studies on Alcohol, supplement no. 14: 14 -22, 2002. 8. McCabe, S. E. "Comparison of Web and Mail Surveys in Collecting Illicit Drug Use Data: A Randomized Experiment." Journal of Drug Education 34(1): 61-72, 2004. 9. Presley, C.; Austin, S. B.; and Jacobs. J. Selecting the Right Tool: A Compendium of Alcohol and Other Drug Assessment and Evaluation Instruments for Use in Higher Education (Washington, D.C.: U.S. Department of Education, Higher Education Center for Alcohol and Other Drug Prevention, 1998). 10. DeJong, W., and Wechsler, H. Methods for Assessing Student Use of Alcohol and Other Drugs (Washington, D.C.: U.S. Department of Education, Higher Education Center for Alcohol and Other Drug Prevention, 1996). 11. Wechsler, H.; Lee, J. E.; Kuo, M.; Seibring, M.; Nelson, T. F.; and Lee, H. "Trends in College Binge Drinking During a Period of Increased Prevention Efforts: Findings from 4 Harvard School of Public Health College Alcohol Study Surveys, 1993 -2001." Journal of American College Health 50: 203-217, 2002. 12. DeJong and Wechsler, Methods for Assessing. 13. Gruenewald, P. J.; Treno, A. J.; Taff, G.; and Klitzner, M. Measuring Community Indicators: A Systems Approach to Drug and Alcohol Problems (London, U.K.: Sage Publications, 1997). 14. Task Force of the National Advisory Council on Alcohol Abuse and Alcoholism, National Institute on Alcohol Abuse and Alcoholism. A Call to Action: Changing the Culture of Drinking at U.S. Colleges (Washington, D.C.: National Institutes of Health, 2002). Resources The U.S. Department of Education's Higher Education Center for Alcohol and Other Drug Abuse and Violence Prevention considers evaluation of programs and policies to be an important component of a comprehensive prevention approach. To help campus administrators develop and execute evaluations of prevention programs, the Higher Education Center offers publications and other materials for free on its Web site at www.higheredcenter.org. Publications College Alcohol Risk Assessment Guide: Environmental Approaches to Prevention by B. E. Ryan; T. Colthurst; and L. Segars This guide outlines methods for identifying and analyzing factors in the campus and community environment that contribute to alcohol-related problems. Scanning and analysis exercises are provided. Finding Out What Works and Why: A Guide to Evaluating College Prevention Programs and Policies by G. Dowdall; W. DeJong; and S. B. Austin Written for campus administrators, this introductory guide outlines procedures for each step of the evaluation process: (1) describing the prevention program; (2) framing evaluation questions; (3) selecting measures; (4) choosing an evaluation design; and (5) reporting results. How to Select a Program Evaluator by L. Langford and W. DeJong This document describes the skills, expertise, and experience to look for when seeking an evaluator; questions to ask when considering an evaluator; and guidance on how to network to find the right person and forge an effective working relationship. Methods for Assessing Student Use of Alcohol and Other Drugs by W. DeJong and H. Wechsler This guide describes methods for gathering and interpreting student survey data on AOD-related problems. Methods for developing questions, drawing a random sample of students, and achieving high response rates are outlined. Selecting the Right Tool: A Compendium of Alcohol and Other Drug Assessment and Evaluation Instruments for Use in Higher Educationby C. Presley; S. B. Austin; and J. Jacobs This compendium reviews issues to be considered when selecting data collection instruments for assessing campus-based prevention programs and describes leading instruments available to the field. More than 20 instrument samples are included. Web Links to Other Materials The Higher Education Center's Web site includes links to a rich compilation of evaluation resources organized by topic, including the following: * Evaluation readiness and building capacity for evaluation * Needs assessment and problem analysis * Program planning, with evaluation in mind * Evaluation design * Protection of human subjects * Measurement and data collection * Qualitative methods, including interviews, focus groups, and observation * Surveys * Geographic Information Systems (GIS) mapping * Archival data * Data analysis and statistics * Reporting and publicizing findings Other Services The Higher Education Center provides training, information, and technical assistance related to evaluation. For further information on these services, visit www.higheredcenter.org. For additional information, contact: The Higher Education Center for Alcohol and Other Drug Abuse and Violence Prevention EDC, 55 Chapel Street, Newton, MA 02458-1060 (800) 676-1730 - TDD Relay-Friendly, Dial 711 HigherEdCtr@edc.org - www.higheredcenter.org This publication was funded by the Office of Safe and Drug-Free Schools at the U.S. Department of Education under contract number ED-04-CO-0137 with Education Development Center, Inc. The contracting officer's representative was Richard Lucey, Jr. The content of this publication does not necessarily reflect the views or policies of the U.S. Department of Education, nor does the mention of trade names, commercial products or organizations imply endorsement by the U.S. government. This publication also contains hyperlinks and URLs for information created and maintained by private organizations. This information is provided for the reader's convenience. The U.S. Department of Education is not responsible for controlling or guaranteeing the accuracy, relevance, timeliness, or completeness of this outside information. Further, the inclusion of information or a hyperlink or URL does not reflect the importance of the organization, nor is it intended to endorse any views expressed, or products or services offered.