How to Select a Program Evaluator by Linda Langford, Sc.D., and William DeJong, Ph.D. [Prevention Updates] This document is also available in Acrobat and Text formats. ------------------------------------------------------------------------ Why Evaluate? Campus-based prevention coordinators are under increasing pressure to evaluate their alcohol and other drug (AOD) and violence prevention programs. The reason boils down to a single word: accountability. Evaluation is especially vital in an era of fiscal constraint. Long-term financial support for prevention work, whether it comes from outside funding sources or is part of a college's regular budget, will be available only if evaluation results warrant it. Evaluation is also important to help program planners revise their programs and policies so that they can focus their efforts on implementing the most effective strategies. Unfortunately, very few campus-based AOD prevention programs have undertaken an evaluation that meets rigorous methodological standards, leaving them vulnerable to attack and fiscal cutbacks. The difficulty is not that prevention coordinators are unaware of the need for evaluation, or that they are worried about their program failing to measure up. Rather, most coordinators do not feel equipped to conduct a rigorous evaluation. One solution to this problem is to hire an outside evaluator. The question, then, is how to find the right one—that is, a qualified evaluator who can meet the program’s evaluation needs. Role of Evaluation in Program Planning and Implementation Because most people associate evaluation with measurement of program results, program planners often do not think about evaluation until after the program is up and running. Instead, evaluation should be planned from the beginning, as the program is being developed. Incorporating evaluation will sharpen everyone's thinking about the program: its mission, its goals, its objectives, and the activities designed to meet those objectives. Used in this way, evaluation planning can be a valuable management tool. Many prevention coordinators have found it useful to view program development and evaluation as an iterative process, with evaluation findings helping inform program modifications. This process requires a system for helping define the problems that need to be addressed (needs assessment) and providing feedback to shape the evolving program (formative evaluation). When seeking to hire an evaluator, prevention coordinators should work with someone who endorses the view that program development and evaluation should work hand in hand. Evaluation Skills and Expertise As in all hiring decisions, when choosing an appropriate evaluator it is important to hire a person whose skills match your needs. At a minimum, a qualified evaluator should be experienced in evaluation design, which involves making decisions about which data to collect, from whom, and at which points in time. Another important component of evaluation is a system for recording the type, frequency, and reach of program activities, from initial planning meetings to actual implementation, or what is called a process evaluation. Conducting a good process evaluation is not as simple as it might seem. Deciding which data are essential, and determining how much data can be collected without becoming unduly burdensome, requires experience and careful thought. Most evaluations will require collection of process data, and an appropriate evaluator should have experience in designing such systems. A formal test of program effectiveness—what is called an outcome or impact evaluation—often involves collecting data from comparison sites without the program. Such an evaluation usually requires external grant support and is best applied to mature programs for which there is preliminary evidence of possible effectiveness. If this is the type of evaluation required, the funding agency might prefer an outside evaluator who works independently of the college or university and can be objective. In the absence of a formal evaluation, an evaluator can help programs develop a system of indicator data (e.g., student surveys, disciplinary records, health services records) to monitor progress in reducing student AOD use and violence. In most cases, the evaluator should have experience with a wide range of quantitative data collection strategies, including telephone or mailed surveys, plus skills in collecting and analyzing official records, referred to as archival data analysis. The evaluator will also need skills in database management and data analysis ("number crunching"). If your evaluation needs include the collection and analysis of qualitative data from one-on-one interviews, focus groups, or field observations, make sure the evaluator has specific skills and experience in that type of work. Many evaluators, including well-known researchers, do not have experience with qualitative methods. Each of the evaluation methods named above entails both designing and then using a protocol or data collection tool. One caution: an evaluator who can design a good tool may not have a lot of experience in using it. For example, an evaluator who can design a good focus group protocol may not be a skilled focus group moderator. Finally, if one of your goals is to involve program participants in designing, conducting, and intepreting the evaluation, you will want to choose an evaluator with experience in what is called collaborative, participatory, or empowerment evaluation. Qualifications and Experience In general, a qualified person will have at least a master's degree in a social science or in public health, have extensive experience as a consulting evaluator, and be willing to work with program staff in a flexible and cooperative manner. Ideally, the chosen evaluator should have experience in evaluating substance abuse or violence prevention programs, including those in higher education. If this individual is familiar with coalition-based programs and the environmental management approach to prevention, so much the better. Having an evaluator who knows the specific content area is less important than having a top-notch evaluator with solid credentials. If necessary, an evaluator can consult publications from the Higher Education Center for Alcohol and Other Drug Prevention to obtain a good introduction to the field of campus-based AOD prevention (see the publications listed on the next page). Some evaluators can meet the need for these diverse skills and experience by working as part of organizations or teams that draw on different individuals with specialized expertise. That option may involve higher costs if the organization includes indirect costs (i.e., administrative "overhead") in its budget. Incentives for the Evaluator If the budget can support it, paying a small stipend or per diem rate will help to recruit an outside evaluator. Hourly rates for individual evaluators vary greatly, but $40 to $80 or more per hour is typical. Outside experts do not always require a lot of money. Assistance from researchers based in univer-sities, policy research firms, or local prevention agencies might be available in exchange for an opportunity to publish a research article, which may serve their personal or institutional needs. Faculty researchers on your campus might be willing to help with the evaluation in order to fulfill their community or institutional service requirements. Creating an advisory evaluation committee that this person can chair might be enough to motivate a faculty member on campus to perform an evaluation. In addition, faculty may be willing to closely supervise master’s or doctoral students who conduct an evaluation study to meet their degree requirements. Networking to Find the Right Evaluator To find the right evaluator, begin by networking with faculty and staff from your school or other nearby colleges and universities who might be able to suggest an appropriate person. Likewise, professional staff based at local and state health departments, research institutes, prevention and treatment services agencies, or other community-based organizations may have recommendations to offer. The Higher Education Center for Alcohol and Other Drug Prevention maintains a database of program evaluators who are available to work with individual campuses (see the Center's Website, www.edc.org/hec/eval). Like other candidates, these individuals will need to describe their experience and provide references for you to be sure they can meet your needs. Alternatively, contact one or more of the following professional associations for assistance in finding experienced program evaluators: * American Psychological Association 750 First Street, NE Washington, DC 20002-4242 (202) 336-5500 website: www.apa.org * American Public Health Association 800 I Street, NW, Washington, DC 20001-3710 (202) 777-2742; Fax: (202)777-2534 E-mail: comments@apha.org Website: www.apha.org * American Sociological Association 1307 New York Avenue, NW Suite 700 Washington, DC 20005 (202) 383-9005; Fax: (202) 638-0882 E-mail: executive.office@asanet.org Website: www.asanet.org A campus with a larger evaluation budget might write up a formal request for proposals (RFP) and send it to outside consulting firms or independent evaluators. Local firms are generally known by reputation, but can also be found in the Yellow Pages. Requiring bidders to propose an evaluation method can help identify the most qualified candidate while also helping program staff be better informed about their evaluation needs. Questions to Ask When considering an evaluator, ask that individual to provide the following information: * Descriptions of past work on similar projects * A summary of past experience with various evaluation methods * History of on-time submission of reports and other "deliverables" * Names and contact information for three references Make sure you interview the person just as you would any potential employee. Discuss any concerns ahead of time and assess the evaluator’s response. Finally, be sure to check references. Forging a Productive Relationship Establishing a good work relationship begins by selecting an evaluator who will be a good fit with existing staff. From there, it is important to discuss the "rules of engagement" that will define that relationship. A good set of rules includes the following: 1. The research relationship will be collaborative. 2. Researchers will provide enough information for program staff to understand the rationale behind specific research decisions. 3. There will be opportunities to discuss and resolve any disagreements about data collection tools or methodology that may arise. 4. The confidentiality of the research subjects will be maintained. 5. Evaluators and program staff will collaborate on how the results are interpreted and discussed in publications and for the newsmedia. 6. Specific evaluation findings cannot be suppressed because of political concerns about the program’s future. Consider making a written agreement ahead of time specifying how the relationship will work. Agree on how and when you will be consulted about the evaluation and who "owns" the data. Who will produce and "own" any written reports or other products? Expect to have to negotiate and compromise. Recognize that evaluators and practitioners come from different professional cultures, with different practices, languages, and customs. Both parties must take time to learn about the other’s culture. There will be times when each other’s behavior is bewildering. Assume good intentions on both sides and keep asking questions until you understand each other’s viewpoint and can reach a compromise. A Final Note A high quality evaluation depends on a commitment by program staff to devote time and energy to the planning and systematic recordkeeping that good evaluation requires. As noted, evaluation should not be viewed as a burden imposed by outsiders, but as an integral part of day-to-day operations. Also keep in mind that, no matter which evaluator is selected, the collaboration will work best if the prevention coordinator is conversant with evaluation concepts. This information is essential for identifying and then working effectively with a good program evaluator. Listed on this page are publications available through the Higher Education Center for Alcohol and Other Drug Prevention that can provide an introduction to evaluation basics. Linda Langford is the associate director for evaluation and assessment and William DeJong is the director of the Higher Education Center for Alcohol and Other Drug Prevention. The Higher Education Center for Alcohol and Other Drug Prevention The Higher Education Center for Alcohol and Other Drug Prevention, established by the U.S. Department of Education, provides nationwide support for campus alcohol and other drug prevention efforts. The mission of the Center is to assist institutions of higher education in developing, implementing, and evaluating alcohol, other drug, and violence prevention policies and programs that will foster students’ academic and social development and promote campus and community safety. The Center offers the following services: * Training and professional development activities * Technical assistance: Resources, referrals, and consultations * Publication and dissemination of prevention materials * Support for the Network of Colleges and Universities Committed to the Elimination of Drug and Alcohol Abuse * Assessment, evaluation, and analysis activities Publications Available Through the Higher Education Center Most of our publications are downloadable from our Website: www.edc.org/hec/. To obtain a printed copy of a publication, e-mail us at HigherEdCtr@edc.org or call us at (800) 676-1730. College Alcohol Risk Assessment Guide: Environmental Approaches to Prevention by B. E. Ryan; T. Colthurst; and L. Segars This guide is designed to help college officials identify factors within the campus environment that contribute to alcohol-related problems. These factors are examined within the context of the public health approach, which emphasizes how the environment shapes behavior. Methods for identifying problems include scanning, analysis, response, and assessment. The publication also contains scanning and analysis exercises and selected resources. This is a great tool for coalitions! 104 pp. Code # 109 Revised 1997 Understanding Evaluation: The Way to Better Prevention Programs by L. D. Muraskin, U.S. Department of Education This handbook describes the "how and why" of program evaluation and outlines the steps involved, working from the premise that many useful evaluations can be conducted by program staff who may not have formal training in evaluation. Although prepared for a general rather than a higher education audience, the information will be valuable to those who need to conduct evaluations required under the Drug-Free Schools and Communities Act (DFSCA) and to those who simply want to find out whether their prevention efforts are yielding results. 98 pp. Code # 905 1993 A College Case Study: A Supplement to Understanding Evaluation by B. Austin This fictitious case study helps prevention specialists, administrators, and others concerned with preventing AOD use on college campuses get a feel for what is involved in setting up an evaluation for a college AOD prevention program and what can be gained from the process. 24 pp. Code # 904 1997 Methods for Assessing Student Use of Alcohol and Other Drugs by W. DeJong and H. Wechsler To develop effective programs and policies for reducing alcohol-related problems on campus, college administrators need to understand fully the nature and extent of these problems at their school. They can achieve this understanding only if they have reliable data on patterns of student alcohol consumption and drinking-related risk behavior. The best way to obtain these data is to conduct an annual survey using a random selection of student respondents. This guide offers a straightforward method for gathering and interpreting student survey data on alcohol-related problems. 48 pp. Code # 104 1995, reprinted 1997 Selecting the Right Tool: A Compendium of Alcohol and Other Drug Assessment and Evaluation Instruments for Use in Higher Education by C. Presley; S. B. Austin; and J. Jacobs This compendium covers the important issues to consider when selecting data collection instruments to monitor progress of prevention efforts and suggest needed policy or program modifications. It also describes the leading instruments used in the post-secondary AOD prevention field. Details provided include application and outcomes, format and administration, costs, validity and reliability, and technical assistance available. More than 20 instrument samples are included. In addition to assessing student AOD use, these instruments cover consequences of AOD use; student, staff, and faculty perceptions of AOD use; environmental factors that encourage use (e.g., tailgate parties) or discourage use (e.g., substance-free housing); and other issues related to the college or university AOD climate. 37 pp. (plus sample instruments) Code # 114 1998 This publication has been funded at least in part with federal funds from the U.S. Department of Education under contract number ED-99-CO-0094. The content of this publication does not necessarily reflect the views or policies of the U.S. Department of Education nor does mention of trade names, commercial products, or organizations imply endorsement by the U.S. Government. Last Update: April 26, 2001 ------------------------------------------------------------------------ [HEC Home]Higher Ed Center -|- About Us -|- Feedback -|- Shortcuts: