Selecting the Right Tool

A Compendium of Alcohol and Other Drug Assessment
and Evaluation Instruments for Use in Higher Education

Cheryl Presley
S. Bryn Austin
Judith Jacobs

A publication of the Higher Education Center for Alcohol and Other Drug Prevention
Funded by the U.S. Department of Education

This article discusses the use of qualitative methods in the evaluation of grant-funded projects.

CONTENTS

• Preface
• Introduction
• Section 1: Choosing the Right Instrument
• Section 2: A Compendium of Measures

Alcohol and Other Drug Surveys
Environmental Assessment
Fraternity and Sorority Assessment
Other Instruments

• Section 3: Annotated Evaluation Bibliography

The Higher Education Center for Alcohol and Other Drug Prevention was established by the U.S. Department of Education in 1993 to assist institutions of higher education in developing and carrying out alcohol and other drug (AOD) prevention programs that will promote campus and community safety and help nurture students' academic and social development.

To accomplish this mission, the Center seeks to increase the capacity of postsecondary schools to develop, implement, and evaluate programs and policies that are built around environmental management strategies. Environmental management means moving beyond general awareness and other education programs to identify and change those factors in the physical, social, legal, and economic environment that promote or abet alcohol and other drug problems.

Clearly, stemming the use of alcohol and other drugs is not something that college administrators alone can achieve. Top administrators, especially presidents, must exercise leadership, but their success will depend ultimately on their ability to build a strong coalition of both on-campus and community interests. The better AOD prevention programs are campuswide efforts that involve as many parts of the college as possible, including students, staff, and faculty. For this reason, the Center emphasizes team-focused training and technical assistance work.

Building coalitions with local community leaders is also key. College campuses do not exist in isolation. AOD prevention planners need to collaborate with local leaders to limit student access to alcohol, prevent intoxication, and support the efforts of local law enforcement. The Center therefore seeks to motivate and train academic leaders to work with local community representatives, while also joining with national organizations that urge local coalitions to increase their outreach to academic institutions.

This publication represents one piece in a comprehensive approach to AOD prevention at institutions of higher education. The concepts and approaches it describes should be viewed in the broader context of prevention theory and the approaches affirmed by the U.S. Department of Education and promoted by the Center in its training, technical assistance, publication, and evaluation activities.

For information on Center services, please contact:

The Higher Education Center for Alcohol and Other Drug Prevention
Education Development Center, Inc.
55 Chapel Street
Newton, MA 02458-1060
Tel.: (800) 676-1730
Fax: (617) 928-1537
Website: http://www.higheredcenter.org/
E-mail: HigherEdCtr@edc.org

The aim of this compendium is to assist administrators, staff, evaluators, and AOD prevention specialists in identifying and selecting feasible, informative, and appropriate evaluation instruments to assess AOD use and monitor program effectiveness on campus.

Since passage of the federal Drug-Free Schools and Communities Act of 1986, colleges and universities have been required to maintain alcohol and other drug (AOD) prevention programs and policies and to conduct biennial reviews of their effectiveness. Partly in response to the federal mandate, institutions of higher education across the country have launched a myriad of ambitious programs and policy changes. Increasingly, school officials have begun to devote a significant amount of staff time and resources to systematic, campus-based assessment of AOD use at their individual institutions. They recognize that the available national data on college AOD use, while informative, are not alone a sufficient basis on which to make programming decisions.

Assessing AOD use and measuring the effects of prevention efforts can be a daunting task, especially when there are a number of programs on campus that promote prevention on multiple levels. A single institution may have in place AOD prevention programs designed to address individual student behavior, interpersonal behavior, perceptions of social norms, institutional change, and community-level change. High demands on resources and staff time also pose challenges to evaluation work. It may be unrealistic to expect every campus to conduct a rigorous, experimental evaluation of its programming and policies. But even so, all schools need to use key indicators to monitor progress and suggest needed modifications. This document will help in either case.

Often the most basic piece of information collected in campus assessments is the prevalence of AOD use. The proportion of students drinking alcohol or using other drugs, the frequency of use, and the amount of use are all critical to understanding the AOD use patterns at a college or university and to monitoring program impact. But there are many other factors that evaluations can assess to help improve understanding of the AOD climate on campus. Selecting the right data collection instruments is a vital component of evaluation. This compendium covers the important issues to consider when making those selections and identifies the leading instruments used in the postsecondary AOD prevention field. The instruments included in this compendium not only assess student AOD use but also cover consequences of AOD use; student, staff and faculty perceptions of AOD use; environmental factors that encourage use (for instance, tailgate parties) or discourage use (such as substance-free housing); and other issues related to the college or university AOD climate.

The chief criteria for inclusion of the instruments in this compendium are that they are

• designed specifically for a postsecondary student population or environment
• well-tested, valid (a term discussed in section 1 meaning that the instrument actually measures what it purports to measure), and reliable (meaning that the instrument provides consistent measurements over time)
• intended for general survey research and not for the diagnosis of alcoholism or alcohol abuse problems in individuals

Section 1 of the compendium presents the primary issues that must be considered when choosing an instrument, which range from format and feasibility to technical concerns about the quality of the measure.

Section 2 has short descriptions of seven instruments selected by a panel of experts in AOD prevention. These tools are among the best available in the field and are designed to cover a variety of areas relevant to campus AOD prevention, including assessment of

• frequency and amount of AOD use
• consequences of AOD use
• student perceptions of AOD use on campus
• fraternity and sorority AOD environment
• faculty and staff perceptions of AOD use
• community coalition involvement in prevention efforts

Section 3 contains an annotated bibliography of resources on research design, statistical analysis, other aspects of evaluation, and additional instruments not included in this compendium.

While informed and careful instrument selection is central to quality evaluation, it is only one piece of a comprehensive evaluation strategy. It is hoped that the information included in this compendium will serve as a valuable resource for administrators, staff, evaluators, and prevention specialists as they develop their evaluation plans and work to increase the effectiveness and efficiency of prevention programming and policy on America's college and university campuses.

• Instruments vary a great deal, not only in terms of the information they assess but also in the way they assess it and how well they do so. Within the AOD prevention field, instruments have been designed to collect data on a range of factors, from student AOD use patterns to perceptions of campus norms to environmental conditions that promote or deter AOD use. Before choosing an instrument it is important to determine what kind of data is needed. If it has not been clearly established which questions about AOD use on campus are the most pressing, there is a risk of selecting an instrument that focuses on irrelevant information or that omits important data. In section 2 of this compendium, the domain, or range of factors covered by an instrument, is described in the summaries under Application and Outcomes.
• A second consideration in selecting an instrument is the target population. Instruments are designed with a particular group of people in mind. For instance, a measure intended for adults may be too difficult for children to understand. Also, a measure surveying college students about drinking on campus would not be appropriate for young adults in the same age range if they were not attending a college or university. All the instruments presented in this compendium are appropriate for the college community, but the intended respondents may be students, faculty and staff, or another subgroup on campus, depending on the measure. In section 2, the target population is described under Informants.
• A third consideration is the format of the instrument. Many use the self-report, forced-choice format typical of standardized tests such as the SAT. Others are self-report but use open-ended questions to allow respondents more freedom in answering. Still others use an interview format, in which a trained staff person conducts a semistructured interview with a respondent or a group of respondents and records their comments. A self-report, forced-choice format is the most feasible for surveying a large number of people but is more restricted than the other formats in the detail of information that can be gathered. The interview format tends to be more labor intensive for staff compared with other survey techniques, but it also has the potential to collect more detailed and complex information from respondents. Format and length determine the time and effort required of respondents and may limit the settings in which an instrument can be administered. For instance, if students are to be surveyed during class time set aside by a cooperating professor, then a 20-minute, forced-choice questionnaire may be most feasible. Information about the format of the instruments is detailed in section 2 under Format and Administration.
• A fourth and arguably the most important consideration is the technical quality of instruments. Experts in evaluation refer to technical quality as the validity and reliability of an instrument. Validity is an indication of an instrument's ability to measure what it is intended to measure. Reliability indicates the consistency with which the instrument provides those measurements. Consider, for example, a ruler, an everyday type of measuring instrument. If a ruler has high validity, then we know that the space between lines marking an inch truly is one inch. If the ruler has high reliability, it will measure one inch as the same amount of distance every time we use the ruler.

Determining validity and reliability is essential to establishing the technical quality of an instrument and ultimately to producing credible evaluation results. Since validity and reliability testing is time consuming, the instrument selection process should always include a careful review of existing instruments that have been fully tested. Section 2 of this compendium includes information on the validity and reliability of each of the featured instruments. Because an understanding of these concepts is so critical to well-informed instrument selection, they are described in greater detail below.

Validity
There are three types of validity used in the field: content validity, construct validity, and predictive validity. In general, validity is not an all-or-nothing characteristic; rather, it is described in degrees from low to high. The more evidence gathered through multiple studies that shows that an instrument performs well in terms of the three types of validity, the more confidence evaluators can have in the measure and, most important, in the study results.

Content Validity.
Content validity refers to whether the questions — called items — included in an instrument cover the whole domain of factors it is intended to address. This form of validity is often determined by agreement among experts. For example, an instrument intended to measure patterns of alcohol use on a college campus would need to include items covering at least three areas: the proportion of students who drink, how frequently they drink, and how much they drink. If a measure addressed only one of these aspects of college student alcohol consumption, experts would consider it to have poor content validity.

Construct Validity.
Construct validity refers to the extent to which an instrument successfully measures a theoretical concept–called a construct–such as anxiety, peer pressure, or perceptions of drinking norms. Scores from one instrument can be compared with scores from others that are intended to measure the same construct to assess whether the instruments perform comparably. Moderate or high correlation between instruments designed to measure the same construct suggests that the instruments are performing as intended.

Predictive Validity.
Predictive validity indicates an instrument's ability to provide meaningful patterns of results. For instance, student performance on the SAT is used by admissions committees to predict academic performance in college. How well SAT scores actually do predict success in college would be an indication of the SAT’s level of predictive validity. Predictive validity is also used to refer to an instrument's ability to discriminate among groups of respondents who would be expected to score differently on a particular measure. For example, a researcher could conduct interviews with students about their alcohol consumption and then divide them into two groups, one made up of heavy drinkers and the other of light drinkers. If the researcher then had the students complete a questionnaire on AOD use, the expectation would be that the heavy drinkers would score differently on the instrument than light drinkers. If the questionnaire could not discriminate between the two groups, it would be considered to have low predictive validity.

Reliability
Reliability, which refers to a measure’s consistency, can be divided into three main types: test-retest reliability, alternate forms reliability, and internal consistency.

Test-Retest Reliability.
An instrument's ability to measure a construct the same way at repeated testing times is referred to as its test-retest reliability. To assess this type of reliability, an instrument may be administered twice to a group of people. If the time in between testing is not too long and if the construct being measured is relatively stable, the scores at the first test would be expected to be very similar to the scores at the second test. For instance, a group of students could be surveyed about their AOD use over the past year and surveyed again two weeks later. Their reports of AOD use over the past year should not have changed much over the two weeks, so if an instrument has a high test-retest reliability, each student's score on the first test should highly correlate with his or her score on the second test.

The magnitude of the test-retest correlation can be represented by a number ranging from 0 to 1, where 1 indicates perfect correlation and 0 represents no correlation. In the context of test-retest reliability, a correlation of .85 or above is generally considered high.

Alternate Forms Reliability.
Reliability can also be established if two alternate forms of the instrument produce highly correlated scores when administered to the same group of people. The alternate forms of the instrument must be comparable, that is they must include the same types of questions in the same format. Evaluators may want to use more than one version of a questionnaire in a situation, for instance, where surveys are repeated in a short time period, raising concern that respondents’ memory of their answers may influence subsequent responses.

Internal Consistency.
Internal consistency is described by the correlations among items within an instrument. If an instrument measures a single construct, all the items would be expected to be highly correlated. Likewise, if an instrument is really measuring two related but distinct constructs, the items might cluster into two groups, where those relating to one construct correlate highly with each other but less so with items in the other group and vice versa. For instance, if an instrument included questions about both binge drinking and consumption of alcohol at meals, scores on all the items about binge drinking would be expected to be highly correlated with each other but perhaps only moderately correlated with items on drinking at meals.

Internal consistency is frequently represented by a statistic called Cronbach's alpha, which also can range from 0 to 1. Cronbach's alpha above .85 is considered high but a value above .70 usually is considered an indication of adequate internal consistency. Another number used to represent internal consistency is the item-to-total correlation, which indicates the correlation — again, ranging from 0 to 1 — between the score on any one item and the total score on the test as a whole. Generally, scores for each item should fall between .3 and .7.

Key Contact:
Core Institute
Student Health Programs
Kesnar Hall MC 6802
Southern Illinois University
Carbondale, IL 62901
Phone: (618) 453-4366
Website: http://www.siu.edu/~coreinst/

Application and Outcomes:
The Core Alcohol and Drug Survey was developed in 1989 to assess the nature, scope, and consequences of alcohol and other drug use on college campuses. This survey was expanded in 1994 to include assessment of students' perceptions and beliefs about alcohol and other drug (AOD) use and AOD-related sexual behavior and violence.

For each behavior measured, an item score can be obtained. Quantity and frequency of use is measured by annual prevalence of use for all drugs and alcohol, 30-day use for all drugs and alcohol, average number of drinks per week, binge drinking within the last two weeks, and change in drinking and drug use in the past 12 months. Problem-related questions measure 19 consequences of drinking and drug use by frequency of occurrence. Another set of questions measures AOD-related incidents of violence. Several items also measure perception of risk, perception of others' use, perceptions of others' feelings regarding use, and perception of campus environment. Social beliefs about the effects of alcohol are measured by 14 items using a forced-choice yes/no format. There are questions that assess other campus variables, such as the extent of participation in campus activities, frequency of refusal behavior, and interference from others’ drinking in a student’s life on or around campus.

Informants:
Postsecondary student population

Format and administration:
The long-form version of the self-report Core Alcohol and Drug Survey includes 39 questions on use and consequences, as well as demographics. The short-form version included 23 questions. Individual questions have between 2 and 19 response options. The most common methods of administration are to mail surveys to a randomly selected subset of students or to administer surveys in randomly selected classrooms. Each method is discussed in the Core Survey User’s Manual, which is distributed upon purchase of the instrument. Colleges can have the results of the survey analyzed by the Core Institute. The long form of the survey takes from 20 to 35 minutes for respondents to complete.

Technical Assistance:
Technical assistance for those administering the Core Alcohol and Drug Survey is available from the Core Institute through a 15-minute video ($25) that describes how to conduct the survey; through the Core Institute’s Website (http://www.siu.edu/departments/coreinst/public_html/index.html);or via telephone.

Costs:
The Core Alcohol and Drug Survey is distributed from and analyzed by the Core Institute. The costs for obtaining the long form of the survey and having the results compiled and analyzed are as follows:

Purchase of each survey copy: $ 0.22
Scanning of each returned survey: $ 0.17
Executive Summary:$20.00
Abbreviated Analysis:$45.00
Disk with Raw Data:$13.00

Use in Evaluation:
The Core Alcohol and Drug Survey is one of the most rigorously developed survey instruments for postsecondary populations. Its original intent was to assess only the frequency and consequences of AOD use. After hundreds of two- and four-year institutions of higher education had used the Core Survey, many requests were received to expand the content of the survey to include other aspects of campus life related to substance issues. Consequently, the survey was expanded to include questions on sexuality, campus violence, institutional climate, perceptions of AOD use, and extracurricular activity involvement. Development of both the original and expanded Core Survey followed strict American Psychological Association (APA) guidelines for test development.

This instrument is not used to diagnose alcohol dependency in individuals but rather to assess the level and impact of alcohol and other drug use on campus. It is a valuable tool for determining how to target populations for prevention programming, designing social marketing and media advocacy campaigns, and assessing the impact of these prevention efforts.

Validity and Reliability:
Considerable evidence supports the validity of the Core Survey. To establish content validity, existing instruments and literature were reviewed to ensure that important aspects and consequences of alcohol and other drug use were adequately covered. A panel of experts then reviewed the items to assess whether they sampled the domain of interest. The level of agreement for item inclusion among the experts was very high (interrater reliability was .90).

Test-retest reliability was estimated using Pearson product-moment correlation coefficients. For items on AOD use and consequences of use, test-retest reliability was high, with the majority of item correlations falling above .80. Items on campus AOD norms showed moderate test-retest reliability, with most correlations falling between .30 and .80. Item-to-total correlations were calculated to assess the internal consistency of the survey. Correlations for the majority of items on AOD use, consequences, and campus norms fell between .30 and .70, as recommended by Henryssen.

Comments:
The Core Alcohol and Drug Survey is easy to administer and has substantial validity and reliability data to support its use with postsecondary populations. Optical scan scoring of the survey is done at the Core Institute, and a report of results is provided to the institution administering the survey, generally within a week. In addition, the Core Institute can provide a computer disk with the institution's survey data in a format that will allow further data analysis.

Another benefit of using the Core Survey is that institutions can compare results with those of similar institutions from across the country that have also used the instrument. Compiled results from other colleges and universities are published in Core Institute monographs every two years and are available on the Core Institute’s Website.

Key Contact:
Core Institute
Student Health Programs
Kesnar Hall MC 6802
Southern Illinois University
Carbondale, IL 62901
Phone: (618) 453-4366
Website: http://www.siu.edu/~coreinst/

Applications and Outcomes:
The Campus Survey of Alcohol and Other Drug Norms was developed in 1997 to assess respondents’ AOD use and their perceptions of campus norms and substance use by other students.

The survey was developed in response to the almost universal finding that students overestimate AOD use by their peers. This misperception is of concern to the extent that students then try to emulate the perceived norm. The first set of questions asks how frequently the respondent believes students at their institution use alcohol and marijuana and other illicit drugs. The second set of questions asks students to estimate the percentage of students who do not drink alcohol and of those who binge drink. Students are asked to report their perceptions of how much students typically consume and of places where alcohol is consumed. There is also a section that elicits respondent attitudes regarding other aspects of drinking and drug use. Finally, respondents are asked about their perceptions of campus policy and policy enforcement.

Informants:
Postsecondary student population

Format and Administration:
The self-report Campus Survey of Alcohol and Other Drug Norms consists of 17 behavioral, perceptual, and attitudinal questions and nine demographic questions. Items range from 2 to 10 response options. Respondents average 12 minutes to complete the survey. The campus survey can be combined with the Core Alcohol and Drug Survey or the Faculty and Staff Environmental Alcohol and Other Drug Survey to elicit a broad view of AOD problems on the campus. The Core Institute is available to optically scan questionnaires and analyze survey results for individual colleges and universities.

Technical Assistance:
Technical assistance for those administering the Campus Survey of Alcohol and Other Drug Norms is available from the Core Institute through a 15-minute video ($25) that describes how to conduct the survey; through the Core Institute’s Website (http://www.siu.edu/departments/coreinst/public_html/index.html); or via telephone.

Costs:
The Campus Survey of Alcohol and Other Drug Norms is distributed from and analyzed by the Core Institute. The costs for obtaining the survey and having the results compiled and analyzed are as follows:

Purchase of each copy: $ 0.27
Scanning of each returned survey: $ 0.17
Executive Summary: $25.00
Cross-Tab Analysis: $30.00
Disk with Raw Data: $13.00

Use in Evaluation:
While the Campus Survey of Alcohol and Other Drug Norms was not introduced on college campuses until 1997, various items on the survey have been used extensively in research and to develop social marketing programs for correcting campus misperceptions regarding AOD use and changing campus norms. The items were based on theory in AOD prevention research.

Validity and Reliability:
Content validity was assessed based on the level of agreement among the experts for item content. There was unanimous agreement among the experts on the choice of items. Many of the items have been evaluated for use in other instruments.

Comments:
The campus survey has a strong basis in theory, and many of its items have been tested in earlier research studies. In addition, results were found to be stable over a four-week test-retest period. The survey’s design enhances ease of administration, scoring, and interpretation of the results.

Key Contact:
Core Institute
Student Health Programs
Kesnar Hall MC 6802
Southern Illinois University
Carbondale, IL 62901
Phone: (618) 453-4366
Website: http://www.siu.edu/~coreinst/

Application and Outcomes:
The Faculty and Staff Environmental Alcohol and Other Drug Survey, developed in 1993, consists of five subscales developed to assess faculty and staff perceptions of AOD problems on campus, awareness of policy and policy enforcement, support for programming efforts to combat AOD problems, awareness of university assessment efforts, and faculty and staff’s perceived ability to identify students who are experiencing a problem and refer them for help. The basis for this instrument was the assumption that faculty and staff attitudes, perceptions, and beliefs about the extent of alcohol and other drug problems are a major influence on the campus environment.

Informants:
Postsecondary faculty and staff

Format and Administration:
The Faculty and Staff Environmental Alcohol and Other Drug Survey consists of 35 items in a forced-choice format, plus seven demographic questions. Responses are marked on an optical scan sheet. The Core Institute is available to analyze survey results for individual colleges and universities. The survey usually takes between 20 to 30 minutes to complete.

Technical Assistance:
Technical assistance for those administering the Faculty and Staff Environmental Alcohol and Other Drug Survey is available from the Core Institute through a 15-minute video ($25) that describes how to conduct the survey; through the Core Institute’s Website (http://www.siu.edu/departments/coreinst/public_html/index.html); or via telephone.

Costs:
The Faculty and Staff Environmental Alcohol and Other Drug Survey is distributed from and analyzed by the Core Institute.

The costs for obtaining the long form of the survey and having the results compiled and analyzed are as follows:

Purchase of each copy: $ 0.19
Scanning of each returned survey: $ 0.13
Cross-Tab Analysis: $30.00
Disk with Raw Data: $13.00

Use in Evaluation:
The faculty and staff survey has been used in faculty workshops and trainings, pre- and posttest evaluations, preintervention assessments, and in the development of media advocacy, social norms, and social marketing programs. The survey provides information about faculty and staff attitudes and their willingness to support prevention programming efforts. Additionally, this instrument can serve as a recruitment tool for faculty and staff involvement in campus efforts to prevent alcohol and other drug problems.

Validity and Reliability:
Research supports the validity of the faculty and staff survey. The level of agreement among the experts on the instrument was .90 in an evaluation of content validity. Some of the items included in the instrument were validated using other instruments such as the Core survey and the National Forum Assessment Guide. The item-to-test correlation is strong, giving more evidence of the instrument’s validity.

Cronbach’s alpha for the instrument as a whole is .81. Cronbach’s alpha for the five subscales are: policy =.75, prevention program awareness = .65, involvement in prevention = .67, assessment = .57, and perception of problem = .62.

Comments:
The faculty and staff survey can be used in a number of ways to provide a more complete understanding of the total campus environment. As one of the most stable elements in the college community, faculty and staff can provide valuable insight into factors affecting student life and the whole campus environment.

Key Contact:
The Higher Education Center for Alcohol and Other Drug Prevention
55 Chapel Street
Newton, MA 02458-1060
Phone: (617) 969-7100; Toll-free: (800) 676-1730
Website: http://www.higheredcenter.org/

Application and Outcomes:
The College Alcohol Risk Assessment Guide (CARA) was developed in 1994 and revised in 1997. This instrument provides campus teams with tools for systematic investigation of the extent of alcohol problems on and off campus and for assessment of the environment that shapes student drinking decisions and adverse consequences. Campuses are encouraged to use the CARA as part of broad-based campus prevention initiatives that involve coalitions and task forces. Unlike previous instruments reviewed in this handbook, CARA does not provide tools for tracking quantitative data regarding alcohol use and consequences. Therefore, the authors recommend the use of CARA in combination with other survey instruments such as the Core Survey.

Informants:
Postsecondary students, faculty, staff, including groundskeepers and housekeeping staff, and other members of the campus and surrounding community

Format and Administration:
CARA is a narrative publication that provides a set of tools to assist campuses in understanding environmental approaches to prevention and in collecting information in a structured way in order to describe and monitor environmental factors in alcohol use. CARAis best used within the context of a prevention coalition or task force. These groups can pick and choose the tools that will most assist them in understanding and implementing environmental prevention measures.

Technical Assistance:
Technical assistance is available through the Higher Education Center for Alcohol and Other Drug Prevention. Center staff and trained Center Associates are available to answer questions regarding the instrument and to offer guidance during its administration.

Costs:CARA is available free from the Higher Education Center.

Use in Evaluation:
CARA analysis tools can be used to collect baseline information for later comparison to postintervention data to assess prevention program impact. Ongoing collection of information on environmental factors using CARA tools will enhance a long-term program evaluation.

Validity and Reliability:
CARA assessment technology is based upon the use of social indicators, which have a long history of acceptance in economics, public health, and many other fields. These tools collect primarily qualitative data and other information useful in program planning and program monitoring. Though not standardized instruments, CARA tools do provide a structured way to collect data.

Comments:
The development of CARA occurred over a three-year period. During this time focus groups, field test groups, a pilot test, and dissemination sessions were held. Over 300 people were trained in the use of these techniques and tools. As shown in a follow-up evaluation, most who were involved in trainings believed the information was valuable to them as prevention specialists.

CARA is aimed at a wide audience on campuses to promote an understanding of the importance of information-based decision making and planning in AOD prevention. The guide's main goal is to shift the focus of campus attention from individuals to the broader environment.

Key Contact:
Center for Alcohol and Addiction Studies
Box G-BH
Brown University
Providence, RI 02912
Phone: (401) 444-1807
Attn: Roger Mitchell, Ph.D.
e-mail: Roger_Mitchell@Brown.edu

Application and Outcomes:
Community coalitions are sometimes organized to support primary prevention of AOD problems on college and university campuses. Several instruments designed by the Community Research and Services Team at the Center for Alcohol and Addiction Studies at Brown University could be revised for use in campus-community prevention programming. These instruments, designed to assess resources, efforts, and efficacy of community prevention coalitions, include three surveys developed in 1989 that are directed at different perspectives on community coalitions: coalition members, coalition leaders, and coalition steering committee members. Additionally, coalition-generated prevention plans can be coded to examine the specifics of planned prevention activities.

The Coalition Member Survey assesses the level of involvement by coalition members, costs and benefits of participation, perceptions of individual skills, perceptions of the efficacy of the coalition, and beliefs about the resulting effects of the coalition members' contributions.

The Coalition Leader Interview assesses perceptions of the strengths and weaknesses of coalition efforts. It assesses satisfaction with current membership, the organizational structure, resources, contacts with other agencies and coalitions, perceived success, and technical assistance needs.

The Executive (or Steering) Committee Member Survey assesses perceptions of the goals and processes for the coalition, perceived benefits of participating in the coalition, difficulties associated with participation, and perceptions of the coalition’s governance and decision-making structure.

The Typology of Prevention Activities provides content categories (e.g. raising awareness, skill building, changing organizational policies) into which trained coders can reliably classify planned programming. This classification scheme can then be used to assess the scope and appropriateness of the types of prevention activities that comprise the programming as a whole.

Informants:
Community coalition members and leaders

Format and Administration:
The Coalition Member Survey is a six-to-eight page, forced-choice, self-report instrument that takes 20 to 25 minutes to complete. It has typically been administered by mail. The Coalition Leader Interview is an in-depth, 16-20 page interview with both forced-choice and open-ended response options, which allows the interviewer to probe as needed. The interview takes about 45 minutes to complete. (A shorter version can be conducted by telephone.) The self-report Executive Committee Member Survey uses a forced-choice response format and takes just 10 to 15 minutes to complete.

The Coalition Member Survey and Executive Committee Member Survey require written responses. They cannot be scanned optically, but the response options for most items easily lend themselves to coding for computer analyses. Responses to the Coalition Leader Interview are recorded by the interviewer, and some open-ended questions require interpretation and coding. For each of the above surveys, subscales can be added or deleted depending upon the interests of the evaluator.

With the typology measure, each prevention activity is coded in one of several specific categories. This allows users to construct a profile that graphically displays where coalitions are devoting their efforts. An additional measure can be calculated to assess the comprehensiveness and scope of the prevention strategies being used.

Technical Assistance:
While no formal technical assistance currently exists, several articles describing how the instruments may be used are available (see footnotes to these pages). In addition, enquiries may be directed to Dr. Mitchell at Brown University.

Costs:
No costs are associated with obtaining the survey instruments. However, the Center for Alcohol and Addiction Studies asks that acknowledgment be given and that outcomes and reports generated through the use of the instruments are shared with the Center.

Use in Evaluation:
The constructs measured by these surveys can be used to assess the efficacy of many types of coalitions. Since coalitions are often made up of individuals holding disparate views, the results of these surveys may provide excellent information for planning, monitoring, and redirecting coalition development.

Validity and Reliability:
Individual scales from the instruments have undergone psychometric development, including principal components analysis and calculation of Cronbach's alphas and correlations with other scales to investigate construct validity. Cronbach’s alphas on most subscales are above .80.

For the Typology of Prevention Activities, the level of agreement among the experts on reliability in assigning activities to categories was at adequate levels (Cohen’s Kappa = .65). In addition, a measure of the scope or comprehensiveness of coalition work was created based on the number of different types of strategies (such as raising awareness of a problem or skill building) used. When comprehensiveness scores were compared to independent ratings by staff, the correlation was r = .51, suggesting moderate convergent validity.

Comments:
These surveys ask important questions that can be used to enhance the efficacy of task forces and coalitions. The information can help close information gaps and address issues of disempowerment and discouragement among task force members. These surveys are appropriate for use in college communities seeking environmental change in AOD use.

Key Contact:
Office of the Vice President for Student Development and Athletics
110 Morrill Hall
University of Minnesota
100 Church Street, SE
Minneapolis, MN 55455
Phone: (612) 626-8127
Attn: Roger Harrold, Ph.D.

Application and Outcomes:
Originally a 1986 University of Minnesota survey, the Greek Experience: A Survey of Fraternities and Sororities was adapted in 1996 to be usable at any college or university. It assesses the problems, attitudes, behaviors, and basic demographics of sorority and fraternity members. This comprehensive instrument also assesses many aspects of the Greek experience, including chapter affairs, academics, personal growth, and alcohol use. Several questions from the Core Alcohol and Drug Survey are used to assess alcohol-related behavior problems. Six questions tap into personal growth, development, and involvement. Revisions to the survey will be completed with the input of national Greek leaders and scholars who will make suggestions for refining the survey. No changes will be made to the Core questions.

Informants:
College sorority and fraternity members

Format and Administration:
The Greek survey is a 63-item, self-report instrument with a mix of forced-choice and open-ended questions. There are 8 items on the survey assessing problems with alcohol. The survey can be administered through mail or in person and takes about 30 minutes to complete.

Technical Assistance:
Technical assistance for administering the survey may be obtained by telephone or through a manual describing how to conduct the survey on campus. In addition, on-site technical assistance in which a Greek adviser may assist schools in planning the administration of the survey will soon be available.

Costs:
The Greek survey is distributed through the Office of the Vice President for Student Development and Athletics and analyzed by Minnesota’s Office of Measurement. Costs associated with the survey are as follows:

Purchase of each copy: $ 0.35
Slicing and optical scanning per survey: $ 0.28
Cross-tabulations report (1,400 tables):$35.00
Operations and systems costs (including disc
with raw data):$70.00
Optional reports on each individual chapter:$30.00

Use in Evaluation:
Many in the campus community, including fraternity and sorority members, have expressed concern about the high use of alcohol by Greek-affiliated students. This instrument can provide good baseline data on fraternities and sororities to aid prevention programming efforts.

Validity and Reliability:
Questions relating to alcohol for this instrument were taken from the Core Alcohol and Drug Survey, which has demonstrated high validity and reliability. (See above section on the Core Alcohol and Drug Survey.) The level of agreement among the experts for some of the other scales in the Greek survey is .90, suggesting high content validity.

Comments:
This survey was developed by Dr. Roger Harrold, who has nearly 30 years’ experience in test development and assessment at the University of Minnesota. Some of the items included in this instrument have been used in longitudinal studies at that institution for as many as 25 years.

An advantage of this survey is the wide range of questions, which cover aspects of the Greek experience left untapped by many other instruments. Ease of administration and scoring is also a benefit.

Key Contact:
IOX Assessment Associates
5301 Beethoven Street
Suite 190
Los Angeles, CA 90066
Phone: (310) 822-3275

Applications and Outcomes:
The Program Evaluation Handbook: Drug Abuse Education was developed in 1988 by IOX Assessment Associates for the Centers for Disease Control and Prevention to assist evaluators of health education programs dealing with alcohol and other drug problems in schools and communities. It presents several measures developed by the authors that may be helpful in assessing various aspects of program effectiveness not otherwise covered in this compendium.

The Friends and Family Survey assesses the perception of support and assistance from those who are closest to the respondent. This can be used in a pre- and posttest situation where there is an intervention specifically designed to change the respondents' perceptions of their social support network. The Systematic Decision Making Survey assesses the respondents' ability to identify steps in a systematic decision-making process. The Ways of Coping Survey measures the respondents’ choices for activities in coping with a variety of difficult situations.

Informants:
Adults

Format and Administration:
These self-report surveys contain from 10 to 20 items with a mix of closed- and open-ended response options. Formulas for scoring are provided with the surveys, so the administrator can score the results easily. The surveys can be administered in a number of settings, including training sessions and prevention programs.

Technical Assistance:
Technical assistance is not currently available.

Costs:
The Program Evaluation Handbook: Drug Abuse Education costs $19.95 plus $5.00 shipping and handling.

Use in Evaluation:
The surveys are designed for evaluation of drug abuse education programs. They are varied and may be used over time to measure changes in specific types of program outcomes.

Validity and Reliability:
These instruments were developed by experts, but there is no data on the technical quality of the measures.

Comments:
A limitation is the lack of reliability and validity data on these surveys; however, the authors of the handbook take the reader through the methods for doing small-scale reliability and validity studies that can be conducted on college and university campuses.

ANNOTATED EVALUATION BIBLIOGRAPHY

Austin, B. A College Case Study: A Supplement to Understanding Evaluation: The Way to Better Prevention Programs. Washington, D.C.: U.S. Department of Education, Higher Education Center for Alcohol and Other Drug Prevention, 1997.

In telling the story of a fictitious college, this case study helps readers get a feel for what is involved in setting up an evaluation of a college AOD prevention program, and what can be gained from such a process. This publication was prepared by the Center as a companion to Understanding Evaluation, which is directed at a broader audience. 24 pp.

DeJong, W., and Langenbahn, S. Setting and Improving Policies for Reducing Alcohol and Other Drug Problems on Campus: A Guide for Administrators. Washington, D.C.: U.S. Department of Education, Higher Education Center for Alcohol and Other Drug Prevention, 1995, reprinted 1997.

Written for administrators at both two-year and four-year institutions of higher education, Setting and Improving Policies for Reducing Alcohol and Other Drug Problems on Campus is especially valuable to the committee assigned to develop and revise alcohol and other drug prevention policies. Other members of the college community who have a special interest in substance use prevention–faculty, students, program directors, campus, security, and other staff–will also find this publication a useful introduction to campus policy setting.

The guide outlines a step-by-step process for establishing or revising policies to prevent and deal with problems arising from student use of alcohol and other drugs on campus. It describes the environmental management approach that can be used to create a safer campus that nurtures students’ academic and social development. 114 pp.

DeJong, W., and Wechsler, H. Preventing Alcohol-Related Problems on Campus: Methods for Assessing Student Use of Alcohol and Other Drugs. Washington, D.C.: U.S. Department of Education, Higher Education Center for Alcohol and Other Drug Prevention, 1995.

To develop effective programs and policies that can reduce alcohol-related problems on campus, college administrators need to understand fully the nature and extent of these problems at their school. This understanding can be achieved only if administrators have credible data on patterns of student alcohol consumption and drinking-related risk behavior. The best way to obtain these data is to conduct an annual survey using a random selection of student respondents.

This guide offers a straightforward method for gathering and interpreting student survey data on alcohol-related problems. The procedure is based on the methodology used in a national college alcohol study conducted in 1993 by the Harvard School of Public Health; it can be easily adapted for use on all campuses. 48 pp.

Muraskin, L. D. Understanding Evaluation: The Way to Better Prevention Programs, Washington, D.C.: U.S. Department of Education, 1993.

Developed with funding from the U.S. Department of Education, Understanding Evaluation describes the how and why of program evaluation and outlines the steps involved, working from the premise that many useful evaluations can be conducted by program staff who may not have formal training in evaluation. The information will be valuable to those who need to conduct evaluations required under the Drug-Free Schools and Communities Act (DFSCA), as well as those who simply want to find out whether their prevention efforts are yielding results. 98 pp.

Ryan, B. E.; Colthurst, T.; and Segars, L. College Alcohol Risk Assessment Guide: Environmental Approaches to Prevention. Washington, D.C.: U.S. Department of Education, Higher Education Center for Alcohol and Other Drug Prevention, 1994, revised 1997.

Designed to assist college administrators in identifying and changing factors within the campus environment that contribute to alcohol-related problems. Factors are examined within the context of the public health approach, which emphasizes ways in which the environment shapes behavior. A series of exercises based on the SARA (scanning, analysis, response, assessment) method are given to assist in identifying specific problems at a given college. 104 pp.

Allen, M. J., and Yen, W. M. Introduction to Measurement Theory. Monterey, Calif.: Brooks/Cole Publishing Company, 1979.

This is an excellent reference book for people who are constructing or using measures. 310 pp.

American Psychological Association. Ethical Principles in the Conduct of Research with Human Participants. Washington, D.C.: American Psychological Association, 1982.

This volume is designed to help researchers understand the rationale and guidelines for using human subjects in research. It is especially useful for handling the ethical issues involved in studying high risk behaviors, such as alcohol and other drug use. 76 pp.

Center for Substance Abuse Prevention. Community Risk Assessment Guide: Teen Drinking Prevention Program. Rockville, Md.: U.S. Department of Health and Human Services, Center for Substance Abuse Prevention, 1995. [NCADI: Inventory Number PHD703]

Methods for conducting a community needs assessment to gather information about alcohol use by underage individuals and the community factors that influence this us are described. 38 pp.

Committee for Developing Standards for Educational and Psychological Testing. Standards for Educational and Psychological Testing. Washington, D.C.: American Psychological Association, 1985.

This book presents standards for construction of tests and is a helpful guide in understanding the ethical issues involved. It is also useful when using a survey for which standard norms have not been established. 99 pp.

Cronbach, L. J., and Furby, L. 1970. How Should We Measure "Change"–Or Should We? Psychological Bulletin 74(1): 68—80.

This thought-provoking article highlights some of the questions researchers face when measuring change and interpreting change scores.

Fetterman, D. M.; Kaftarian, S. J.; and Wandersman, A. (eds.). Empowerment Evaluation: Knowledge and Tools for Self-Assessment and Accountability. Thousand Oaks, Calif.: Sage Publications, Inc., 1996.

Empowerment evaluation, the use of evaluation concepts, techniques, and findings to improve programs and foster self-determination among program participants, is discussed. 411 pp.

Goldfield, N. G., and Pine, M. Measuring and Managing Health Care Quality: Procedures, Techniques, and Protocols. Gaithersburg, Md.: Aspen Publishers, 1996.

This is a fairly detailed but very readable and timely handbook addressing the complexities of process evaluation, continuous program improvement, and community involvement. 213 pp.

Kumpfer, K. L.; Shur, G. H.; Ross, J. G.; Bunnell, K. K.; Librett, J. J.; and Millward, A. R. CSAP Technical Report 8: Measurements in Prevention: A Manual to Selecting and Using Instruments to Evaluate Prevention Programs. Rockville, Md.: U.S. Department of Health and Human Services, Center for Substance Abuse Prevention, 1993. [NCADI: Inventory Number BK213]

Guidelines for identifying measures and instruments for the evaluation of an alcohol and other drug prevention program that are relevant to both the program and the target population are discussed. 204 pp.

Linney, J. A., and Wandersman, A. Prevention Plus III: Assessing Alcohol and Other Drug Prevention Programs at the School and Community Level: A Four-Step Guide to Useful Program Assessment. Rockville, Md.: U.S. Department of Health and Human Services, Office for Substance Abuse Prevention, 1991. [NCADI: Inventory Number BK188]

A step-by-step approach to assessing alcohol and other drug prevention programs at the school and community level is described. 461 pp.

Mills-Novoa, B. "The Use of Qualitative Methods in the Evaluation of Grant-Funded Projects," in The Grantseeker's Guide to Project Evaluation, Second Edition, edited by Jacqueline Ferguson. Alexandria, Va.: Capitol Publications, 1997: 63—69.

A publication of the Higher Education Center for Alcohol and Other Drug Prevention

Funded by the U.S. Department of Education

For further information contact:
The Higher Education Center for Alcohol and Other Drug Prevention
http://www.higheredcenter.org/
(800) 676-1730
Fax: (617) 928-1537
HigherEdCtr@edc.org

Published 1998
The Higher Education Center for Alcohol and Other Drug Prevention
Education Development Center, Inc.
55 Chapel Street
Newton, Massachusetts 02458-1060

Last Update: July 11, 2000