Your ability to capture and communicate evidence on the benefits of your conflict management program is one of the best ingredients for building wider support and obtaining future funding. On Day 4 we introduce you to basic concepts of program evaluation and give examples of how these can be used effectively. We also discuss the importance of research-proven practices for conflict management programs. Should the question of corroborating evidence of such programs’ value come up during the planning and early implementation stages, we provide a review of the most recent research in this area for you to cite in your response.

The Need for Evaluation

“Issues of evaluation are issues of strategy. Purity of method is no virtue. The best strategy matches the methods to the questions being asked.”
 – Michael Quinn Patton, Social Scientist/Anthropologist

Mary’s story is a common one. Teachers and administrators know that program evaluation is a necessary and expected part of developing a conflict management program; yet, they often don’t do it. Evaluation may seem complicated and costly in terms of time and resources.

In this Day 4 content, we will concentrate on helpful tips that you can use to plan your CRE program evaluation. We begin with a basic overview of some good reasons to do program evaluation. Then we introduce two basic orientations to program evaluation: process evaluation and outcome evaluation. And later, we provide a review of research that has been conducted on CRE programs – research that helps bolster your program by proving that similar programs have had benefits in similar situations.

The main reason to do program evaluation is to see if the program works. But that is a deceptively simple statement. When you actually contemplate a program evaluation you are assuming what “works” means and why it may be important to you. Let’s consider some of the typical reasons that program evaluation is wanted and warranted.

Complying With a Mandate:
It is becoming increasingly common for school boards, and external funders to require that programs be evaluated in order for them to be approved and funded, as Mary learned in the opening story. The federal government has mandated that schools receiving money from the Safe and Drug Free School areas must have a method of evaluating their program and present that plan in order to obtain funding. Many state and local agencies or private foundations are moving to a stronger stand on required evaluation. And, even when the resources for a program are not obtained externally, decision-makers are usually interested in having proof that the resources devoted to your conflict management program were well spent.

Making Sure the CRE Program Is Working As Planned:
When a school decides to implement a conflict management effort, whether it is a peer mediation cadre program or a peaceable school initiative, there are needs that have been identified or goals that have been articulated. The program is developed to meet those needs or achieve those goals. One of the most compelling reasons to evaluate a program is to determine whether those needs have indeed been addressed or whether the goals have been achieved. Assume that your elementary school started a program because there had been a sudden outbreak of fighting during recess and lunch. The need driving the program was to reduce the amount of fighting and the goal was to see at least a 50% reduction in physical altercations between students. Evaluating your program would examine, at least in part, the extent to which the conflict management was related to the amount of fighting among students. Although it may seem simple, it is important to carefully consider goal setting for conflict management programs and how that is linked to program evaluation that monitors the achievement of goals.

Improving a Program by Identifying Strengths and Weaknesses:
A good reason to do program evaluation is to be able to identify things that are working and things that are not in order to know what to repeat and what to fix. Sometimes we approach program evaluation with an eye for the negative, or the things that aren’t what we’d like to see. It’s important to remember that assessment of both strengths and weaknesses are helpful. In terms of strengths, a program evaluation may show you that one reason you witnessed the hoped-for reduction in fighting was that teachers were very committed to the program and kept reinforcing the ideas to students whenever possible. Your program evaluation may suggest that program effectiveness was due to the quality of assistance gained from outside trainers or the quality of their curriculum. By identifying these strengths you can decide how to maintain teacher commitment and enthusiasm or decide to retain the same trainers and curriculum as the program continues.         

A good program evaluation will help you identify strengths that were expected as well as those that were not. Sometimes, the reasons a program works are unanticipated and program evaluation helps us recognize and perpetuate those. For example, in one middle school that implemented a whole school program, the school saw a very strong increase in school climate. The program evaluation showed that increase, but also helped identify the role that non-teaching assistants (NTAs) had on the increase. In this school, NTAs were parents or community members who volunteered to act as aids, hall monitors, lunchroom attendants, etc. The program evaluation found that the NTAs were talking very favorably about the conflict management program to other parents and community members outside of the school. This support in turn, increased others’ interest in the program and their complements to the teachers and administrators about the program. The teachers and staff felt appreciated and rewarded, which increased their sense of importance and the general climate in the school. Prior to the program evaluation, the NTAs had not been identified as an important link in the overall effort. After the program evaluation, they were seen as a source of strength and a “best practice” of involving NTAs in training and program development was emphasized.

Expanding a Program:
Program expansion often follows an initially positive program evaluation. The program is expanded by introducing it into new schools if the effort is at a district level. Or, it can be expanded by adding more components to the program. For example, program evaluation can provide a good estimate of the staff support necessary for maximum impact of the program. This information is crucial for effective planning and resource allocation in new programs.  If the role of NTAs has proven to be a strength of successful conflict management efforts, strategic planning for new programs may first attend to shoring up the NTAs or encouraging their commitment to the program before implementation. In any expansion effort, whether in the same school or across schools, program evaluation provides the road map for success. It offers the opportunity to learn from previous mistakes, to preserve best practices, and to enhance the efficiency of the process.

Gaining Additional Resources:
People love a winner. This is an old maxim, but a true one when it comes to decisions about where to put already scarce resources. In education, there is little reason to throw good money after bad, especially when a lot of good programs vie for limited funding. Thus, one of the main reasons to do program evaluation is to make the strongest case for why you should continue to receive support for your efforts.            If you have solid program evaluation you can provide this evidence and prove that you have the competence to continue overseeing the processes and outcomes of the conflict management program.

Choosing Your Program Evaluation Focus         

“Research is formalized curiosity. It is poking and prying with a purpose.”
Zora Neale Hurston (1903 - 1960) US dramatist, author

Good program evaluation answers the questions “what happened?” ”how did it happen?” and “why did it happen”?  Program evaluation has two parts – process evaluation and outcome evaluation. Process evaluation is concerned with the process of implementing the program and whether that process is enhancing or inhibiting the success of the program. Outcome evaluation focuses on changes that have happened because of the program.

Process Evaluation
What are some of the processes that one might be interested in monitoring in conflict management? Although the processes may differ depending upon the program model being used, all programs will have some concerns in the following areas.

Assessment Process: The more elaborate the program model the more complex the assessment process should be.

• How well were the needs for the program assessed prior to program development and implementation?
• Were key stakeholders identified?
• Were all relevant voices heard?
• Were needs clarified and well articulated?

Planning Process: Planning processes are critical to program success. Strategic planning insures that you are projecting difficulties and planning for them instead of being caught unaware.

• What planning processes were used to identify program goals and objectives?
• What planning processes were used to secure needed resources?
• What planning processes were used to develop an implementation strategy?
• Did planning consider how the program would be integrated into existing activities?
• For more complex program models, did planning deal with problems of coordination between program components?

Orientation Process: Once the assessment and planning is completed, the program needs to be introduced to the teachers, staff and students at the school. In most cases, some of those people have already been involved in the assessment and planning processes. Process evaluation can focus on how the orientation was accomplished.

• How extensive was the explanation of program content and purpose?
• How was it presented?
• To whom was it presented and in what forum?
• What processes, if any, were used to secure and assess the degree of resistance or commitment to the program?
• How was feedback handled and suggestions for change negotiated?

Selection Process: This is the issue of who is chosen to directly participate in the conflict management program and what the impact of those participation decisions are for the program and school.

• Was participation in the program voluntary?
• In peer mediation efforts, were students allowed to self-select as peer mediators or were students selected by others and expected to perform regardless of their individual desires?
• In curriculum integration activities used in peaceable classroom or peaceable school models, how were students and/or classes selected?
• Did teachers or students have a chance to refuse participation?
• In whole-school/community models, how were community members and parents selected to participate?
• What processes were used to recruit their participation and how well did these work?
• Were interested parties not allowed to participate for some reason?

Training Process: All conflict management program models use some form of training.

• When the program emphasizes or is limited to a mediation component, the usual approach is to have outside trainers from a mediation organization provide training to the students who will serve as peer mediators and the staff who will help coordinate the project.
• When the program model involves teaching foundational abilities or conflict education curriculum in classrooms, the usual approach is the train-the-trainer model. Here, outside trainers or experts in the curriculum teach teachers how to instruct and work with them on lessons and lesson plans.
• In programs that involve parent and community members receiving training, the training process may use either or both of the above approaches. In all training processes there are questions about the nature of the training and the adequacy of the training.
• How long did the training last?
• Who delivered it?
• Were the trainers qualified?
• What kinds of pedagogical approaches were used?
• What instructional or supporting materials were involved?
• What was the quality of those materials?
• What was student/staff/parent/community member reaction to the training?

Program Implementation Process: Up to this point, all activity has been preparatory. Now, the school has been oriented, the participants have been selected, the training has taken place, and the program is ready to be implemented. The process of implementation itself differs considerably by program model.

In peer mediation programs the program implementation process involves issues of program publicity, referral to mediation, utility of mediation, and linkage to other disciplinary activities. Process evaluation also can look at how the program is publicized and with what effect.

• What procedures are used to refer cases to mediation and how well are teachers, staff and students using the referral mechanisms?
• How do issues of referral and publicity relate to how the program is used?
• What processes are used to link mediation to other counseling or disciplinary actions in the school?
• What happens if mediation is not successful; how is the dispute then handled and how are those decisions made?

In curriculum integration models, process evaluation is especially important. It can explain how the curriculum was actually integrated in the classroom situation and the factors that influenced the integration process.

• What curriculum is being used?
• How was the curriculum selected and why?
• What kind of training has been provided to help teachers use the curriculum in an ongoing manner?
• How comfortable are teachers with using the curriculum?
• What changes, if any, were made to the curriculum to better meet student needs?
• How often is the curriculum being used?
• Are teacher rewards or accountability structures linked to appropriate use of the curriculum?
• How well are teachers using a team process to coordinate and orchestrate the activity across classes or grades?
• How are students reacting to the curriculum?

In whole-school models, especially those that have multiple components and involve external and internal groups, process evaluation allows emphasis on program development and coordination. In terms of development, process evaluation helps you explain the stages that the school went through in developing and implementing the various components. Difficulties or successes in one stage can be linked to the progress in another stage. Issues of coordination, especially when school programs involve a strong link to community, are particularly important to document.

• How were parents coordinated into the program?
• How involved are parents being? Allowed to be?
• What community members and organizations are involved?
• What is the parents’ and community members’ understanding of their role in program?
• How do community CRE programs linked to the school efforts impact what can and is being done in the school (or by the students?)
• In what ways did the interaction between community and school facilitate program success in each venue?

Program Maintenance Process: If a program is long-lasting it will have to face the issue of turnover in personnel and participants. Schools that plan for the turnover in a manner that protects the program are able to weather the change. Unfortunately, many programs that make excellent process in all other areas were destroyed when turnover issues are poorly handled. Some questions that can be asked of program maintenance include the following:

• What is the degree of turnover that is experienced in the program?
• How are the selection, initiation and orientation of new members or participants handled?
• How are record-keeping procedures used to maintain a working history and continuity of the program?
• Are program procedures and processes manualized?

Context/Environmental Factors: No matter how well a school plans for and tries to oversee its conflict management program, the reality is that the larger context or environment can influence what happens, sometimes beyond our control. When we talk about context or environment we recognize that there are multiple layers that could be considered. But for the purposes of process evaluation, we suggest attending to at least four.

The school context involves anything happening in the school that is outside the specific operation of the program. It could be the loss of an important leader or administrator that raises uncertainty about the future for the program.

The district context involves changes at the level of the school district and how that impacts school activity. A simple example is the case of a labor action at the district level that affects teacher availability and willingness to participate in an activity like conflict management.

The community context concerns the pressures and supports that are present in the surrounding neighborhood, community, or city in which the school is located. For example, a city may be undergoing cultural or racial tensions that increase incidents of violence and raise awareness for the need for constructive conflict resolution. In that environment, school programs may experience a resurgence of support and resource allocation.

Finally, at the larger societal level, there may be events that achieve statewide or national visibility that impact a school program. Although the influence may be indirect, it is nonetheless present. We all remember the terrible shooting at Columbine High School in Littleton, Colorado. Following that incident, there was a strong increase in utility of and support for conflict management efforts throughout the country. Similarly, when Attorney General Janet Reno proclaimed violence prevention and conflict management a critical need in our schools, her support became a rallying cry that educators found effective for arguing for program development and expansion.

“True genius resides in the capacity for evaluation of uncertain, hazardous, and conflicting information.”
Winston Churchill (1874 - 1965) English statesman, author

Outcome Evaluation
Outcome evaluation focuses on whether and to what extent specific, tangible goals and objectives established for the program are achieved. The emphasis is on the outcomes of the program and your ability to document them. You have two basic goals in outcome evaluation:

(1) to document what happened in terms of utility or frequency (as opposed to how it happened in process evaluation) and

(2) to document what changed as a result of the program. In the first area of outcome evaluation, people attend to questions about the extent of involvement or activity. For example, how many students were actually trained? How many parents attended the workshop sessions? How many cases went to mediation? How many cases were resolved in mediation? How many times were student mediators linked with community mediators in community mediation situations? 

In the second area of outcome evaluation, the notion of proving change requires that you have some way to compare what the situation was before the program with what happened during and/or after the program. Most of us are familiar with pre-test and post-test designs and the use of control groups that are necessary for this kind of outcome evaluation.

The types of outcomes that can be evaluated are as varied as the types of programs that exist. However, we can talk about five general kinds of outcomes that most programs are interested in at some level.

Skills/Abilities Learned: As mentioned in Day 1, conflict management teaches foundational abilities necessary for the enactment of constructive conflict management. Thus, outcome evaluation may focus on questions about how well children learned the skills of active listening, perspective-taking, empathic response, generation of alternatives, anger control, etc. How well do mediators learn the actual process of mediation? How well do parents learn the process of emotional coaching and anger management? The key is to clearly specify the skills and abilities that you are trying to teach in the program and to make sure that these are being evaluated in terms of how well the program developed these skills or abilities in participants.

Attitudes Changed: Many conflict educators are interested in helping students adopt more pro-social attitudes. Depending on the emphasis on social justice concerns, the degree to which a social justice orientation is developed through exposure to the conflict management may be a critical outcome to be monitored. If the guiding motivation is violence prevention, the overriding outcome of concern may be attitudes toward violent and aggressive action or the tendency to make hostile statements about another that may escalate conflict. For whole school programs, an outcome of interest is usually school climate, or the attitudes that teachers and students have about the school. Evaluators are often asked to document how school climate changes over the course of the program implementation. 

Behaviors Changed: Given the link between conflict management and violence prevention efforts, an important outcome to be evaluated for many people is the degree to which violent behavior is decreased as a result of the program. How much less likely is a student to actually fight as a result of training? How many fewer suspensions or expulsions have occurred in the school since the program has been in effect? Have truancy rates dropped or increased? Are students using fewer disruptive behaviors in the classroom? Are teachers holding fewer disciplinary conferences with parents? Are students behaving more collaboratively in home or community settings?

Program Utility: Questions of program utility have to do with the extent to which conflict processes taught have actually been used in the school. In many programs this is the most common form of outcome evaluation. As mentioned above, this focus often assesses how frequently mediation is used, by whom, and with what outcome. Program utility issues are most germane to mediation programs or to the mediation component in peaceable classroom or peaceable school models.

Resources Created: Sometimes outcome evaluation can focus on resources that are created as a central or peripheral purpose of the conflict management program. These resources fall into three categories:

• First, there are the tangible economic resources. For example, sometimes a program outcome is receiving a financial award based on effectiveness or securing additional funding for continued work.
• Second, there are the instructional products. Training manuals and instructional material developed or modified for use in the program can be seen as valuable outcomes.
• Finally, there are relationships and infrastructures that are formed. Especially in programs that link school and community efforts, an important area of outcome is how well the program helped develop relationships between the school and external partners that can serve as the foundation of a larger infrastructure for further programmatic development or expansion.

 

Clarifying CRE Program Goals with Stakeholders

In Day 1 of this course, we introduced you to a list of goals for CRE programs. Here, we use an elaborated list of goals to help guide conversations with your stakeholders. These conversations are very important BEFORE you conduct an evaluation. If you are unclear about what you are trying to achieve, you’ll probably waste valuable time and money in an evaluation without a clear target.

As you review this list, begin thinking about which goals are clearly high priority for you and your school.

Create a Safe Learning Environment

• Decrease incidents of violence
• Decrease conflicts between groups of students; particularly intergroup conflicts based on racial and ethnic differences
• Decrease suspensions, absenteeism and drop out rates related to unsafe learning environments

Create a Constructive Learning Environment

• Improve school climate
• Improve classroom climate
• Promote a respectful and caring environment

Improve Classroom Management

• Reduce the time teachers spend on disciplinary problems in the classroom
• Increase use of student-centered discipline

Enhance Students’ Social and Emotional Development

• Increase perspective taking
• Develop problem-solving abilities
• Improve emotional awareness and emotional management
• Reduce aggressive orientations and hostile attributions
• Increase students’ use of constructive conflict behaviors in schools and in home and community contexts

Create a Constructive Conflict Community

• Increase parental and community involvement in school affairs
• Link school CRE with larger community CRE efforts
• Develop more peaceful communities

CRE programs often have multiple goals. It is most helpful when you look at the program goals before implementing the program and the evaluation. If you are just beginning to plan your CRE program, the more time you spend clarifying your goals, the better off you’ll be. If you have already implemented a CRE program or are thinking of expanding your CRE program, considering program goals is still a very valuable step. And in all cases, knowing your goals becomes a blueprint for how and what you want to evaluate.

 

Assessing Resources

Now that you have an evaluation plan, you know what you want to find out.  Next, you need to take a realistic look at the resources you have to conduct the evaluation.

Information Resources: When you are planning your program evaluation, consider the extent to which you have available information about related research in the area. In many cases evaluators may spend time and energy re-inventing the wheel because they are not aware of other research or program evaluation that has been done.  For example, say your school wants to use a training curriculum that has not been used in the school before. Some stakeholders may have a strong interest in having program evaluation that assesses the students’ reactions to this curriculum. Before deciding to devote energy to this task, see whether other schools in your district, or other schools with similar student populations, have used this curriculum and reported on student satisfaction. In many cases, questions have already been answered, relieving you of the responsibility to repeat the effort. That will free you to spend resources on questions that are important and unanswered. In the last part of this Day 4 material we have a section on a review of recent CRE research to help you decide whether evidence of your program’s success may already exist and can be used to bolster your evaluation.

Data Availability: You should also find out whether data are available or can be made available to assess program impact. For example, if you are interested in seeing whether the conflict management program reduced violence significantly you will need to have baseline data of violence before the program begins. Do you know whether the school or district keeps this data on hand?

Commitment of Stakeholders: Now we are talking about the willingness of the stakeholders to participate in program evaluation processes. This may seem a given, but anyone who has done program evaluation can tell you it is one of the most difficult factors. The reality is that people get busy or do not provide you with the information they once promised--information that is critical to your program evaluation. Talk with stakeholders about the kinds of support they will give, the kinds of information they will provide and the consequences of not getting it. If stakeholders are reluctant initially, you can almost be certain they will not be cooperative when you need the information.

Program Evaluation Experts: Do you have access to program evaluation experts who can serve as mentors or guides during your initial program evaluation experiences? Often area universities or school districts have people on staff with this expertise.

Staffing: Do you have people in the school who are willing to act as program evaluators? This can be more time consuming than expected. Do you have people who can help prepare surveys, collect data, record data, read reports, analyze data, and write reports? The need for clerical kinds of support is greater in process evaluation than outcome evaluation. But the need for expert aid (for statistical analysis) is more pronounced in outcome evaluation.

Money: Do you have the money to pay for research support, office supplies, copying costs, etc.?

Time: Do you have the time to do the evaluation? Does the school staff have time to devote to it? Do teachers have time to allow data collection in their classes?

Technology: If you are doing statistical analyses, or are planning on using graphs in your report, do you have the necessary computer software?

In an effort to help CRE program evaluation and to provide some of the resources indicated above, the document “Looking for Success: Evaluating Your CRE Program” provides a detailed description of the six step process for CRE program evaluation. It also provides proven qualitative and quantitative measures for use in evaluating CRE program success. You can download this document for free from the following web site (http://disputeresolution.ohio.gov/schools/evaluatingcrep.htm).

Research Proven Benefits

The No Child Left Behind Federal Legislation stresses the need for educational programming to be research based.  Trying to choose a program and trainers may seem overwhelming, but there are many resources available to assist you.  In the last decade the research has shown that when effective strategies are used to teach staff and students the skills and concepts of classroom management, these skills can also be applied throughout the school community and can lead to reductions in discipline challenges, improved academic achievement, and a more positive overall school climate.  Below are several documents that have reviewed the research on various programs and include some of the programs that are approved for use with federally funded grants.

Attached is a research review article from Conflict Resolution Quarterly published in 2004 that summarizes relevant research in CRE. In this document there are also references to a variety of other literature review articles on specific programs (like Teaching Students to be Peacemakers) that may be of interest to you.

If you are thinking of making a presentation and would like resource materials concerning current research, the following links will provide you with an annotated bibliography and an accompanying PowerPoint presentation summarizing CRE effectiveness in terms of CRE goals discussed in this course. A brief summary of the research in terms of goals is provided here:

Goals for CRE and Evidence of Effectiveness

• Enhances Students’ Social and Emotional Development
• Strong Evidence That CRE:
• increases perspective-taking
• increases empathy
• improves emotional awareness and management
• reduces aggressive orientations and hostile attributions
• increases use of constructive conflict behaviors
• Creates a Safe Learning Environment
• Strong Evidence That CRE:
• decreases anti-social behavior that leads to violence
• decreases conflicts between groups of students
• decreases suspensions, absenteeism, and drop out rates
• Need More Evidence That CRE:
• decreases incidents of violence – although there is some support for this, CRE programs are not designed to decrease violent conflict and weapons related conflict. Thus, some reviews of research suggest that there is limited evidence that CRE has significant impact on aggravated assault and weapons related assault.
• Create a Constructive Learning Environment
• Strong Evidence That CRE:
• improves school climate
• improves classroom climate
• increases academic achievement
• Need More Evidence That CRE:
• improves teacher/administrator/student relationships – qualitative and anecdotal evidence suggests that this may be one of the benefits of CRE, mainly through creating a healthy and positive school climate, but the outcome evidence has not been a focus in the research
• increases valuing of diversity and practice of tolerance – again, there is certainly anecdotal evidence that this is the case, but no quantitative studies
• promotes a respectful and caring environment – and the same for this
• Create a Constructive Conflict Community
• Need More Evidence That CRE:
• increases parental and community involvement
• links effectively with larger community CRE efforts
• develops more peaceful/peaceable school community

 

 

Discussion Questions

Please think about the questions below and share your responses with colleagues.

How might program evaluation help you in starting or maintaining your CRE program?

What kinds of evidence seem most persuasive to Administrators? Parents? Funders? Other teachers?

What resources would be most helpful to you in planning and conducting your CRE program evaluation?