Managing and Resolving Conflicts Effectively in Schools and Classrooms

Day 4: Best Practice - Data and Research

Your ability to capture and communicate evidence on the benefits of your conflict management program is one of the best ingredients for building wider support and obtaining future funding. On Day 4 we introduce you to basic concepts of program evaluation and give examples of how these can be used effectively. We also discuss the importance of research-proven practices for conflict management programs. Should the question of corroborating evidence of such programs’ value come up during the planning and early implementation stages, we provide a review of the most recent research in this area for you to cite in your response.

The Need for Evaluation

“Issues of evaluation are issues of strategy. Purity of method is no virtue. The best strategy matches the methods to the questions being asked.”
 – Michael Quinn Patton, Social Scientist/Anthropologist

Mary Williams is the coordinator of her middle school’s peer mediation program. She’s coordinated the program for the past five years and has received some release time and a small budget of about $3,000 per year for training and supplies. While the program has seemed successful and well used, Mary has never had the time or skill to conduct an evaluation to prove what the program is accomplishing. Until this year she didn’t feel the need to do this because the program’s continuation seemed certain. And, Mary felt her time was better used keeping the program running than trying to evaluate it. Yet, last week, Mary’s principal asked her in for a sobering talk. The principal explained that the school is facing serious funding cuts and they need to justify every dollar spent. She asked Mary to give her a brief report about how students are using the peer mediation program and how they are benefiting from it.

Mary’s story is a common one. Teachers and administrators know that program evaluation is a necessary and expected part of developing a conflict management program; yet, they often don’t do it. Evaluation may seem complicated and costly in terms of time and resources.

In this Day 4 content, we will concentrate on helpful tips that you can use to plan your CRE program evaluation. We begin with a basic overview of some good reasons to do program evaluation. Then we introduce two basic orientations to program evaluation: process evaluation and outcome evaluation. And later, we provide a review of research that has been conducted on CRE programs – research that helps bolster your program by proving that similar programs have had benefits in similar situations.

The main reason to do program evaluation is to see if the program works. But that is a deceptively simple statement. When you actually contemplate a program evaluation you are assuming what “works” means and why it may be important to you. Let’s consider some of the typical reasons that program evaluation is wanted and warranted.

Complying With a Mandate:
It is becoming increasingly common for school boards, and external funders to require that programs be evaluated in order for them to be approved and funded, as Mary learned in the opening story. The federal government has mandated that schools receiving money from the Safe and Drug Free School areas must have a method of evaluating their program and present that plan in order to obtain funding. Many state and local agencies or private foundations are moving to a stronger stand on required evaluation. And, even when the resources for a program are not obtained externally, decision-makers are usually interested in having proof that the resources devoted to your conflict management program were well spent.

Making Sure the CRE Program Is Working As Planned:
When a school decides to implement a conflict management effort, whether it is a peer mediation cadre program or a peaceable school initiative, there are needs that have been identified or goals that have been articulated. The program is developed to meet those needs or achieve those goals. One of the most compelling reasons to evaluate a program is to determine whether those needs have indeed been addressed or whether the goals have been achieved. Assume that your elementary school started a program because there had been a sudden outbreak of fighting during recess and lunch. The need driving the program was to reduce the amount of fighting and the goal was to see at least a 50% reduction in physical altercations between students. Evaluating your program would examine, at least in part, the extent to which the conflict management was related to the amount of fighting among students. Although it may seem simple, it is important to carefully consider goal setting for conflict management programs and how that is linked to program evaluation that monitors the achievement of goals.

Improving a Program by Identifying Strengths and Weaknesses:
A good reason to do program evaluation is to be able to identify things that are working and things that are not in order to know what to repeat and what to fix. Sometimes we approach program evaluation with an eye for the negative, or the things that aren’t what we’d like to see. It’s important to remember that assessment of both strengths and weaknesses are helpful. In terms of strengths, a program evaluation may show you that one reason you witnessed the hoped-for reduction in fighting was that teachers were very committed to the program and kept reinforcing the ideas to students whenever possible. Your program evaluation may suggest that program effectiveness was due to the quality of assistance gained from outside trainers or the quality of their curriculum. By identifying these strengths you can decide how to maintain teacher commitment and enthusiasm or decide to retain the same trainers and curriculum as the program continues.         

A good program evaluation will help you identify strengths that were expected as well as those that were not. Sometimes, the reasons a program works are unanticipated and program evaluation helps us recognize and perpetuate those. For example, in one middle school that implemented a whole school program, the school saw a very strong increase in school climate. The program evaluation showed that increase, but also helped identify the role that non-teaching assistants (NTAs) had on the increase. In this school, NTAs were parents or community members who volunteered to act as aids, hall monitors, lunchroom attendants, etc. The program evaluation found that the NTAs were talking very favorably about the conflict management program to other parents and community members outside of the school. This support in turn, increased others’ interest in the program and their complements to the teachers and administrators about the program. The teachers and staff felt appreciated and rewarded, which increased their sense of importance and the general climate in the school. Prior to the program evaluation, the NTAs had not been identified as an important link in the overall effort. After the program evaluation, they were seen as a source of strength and a “best practice” of involving NTAs in training and program development was emphasized.

Expanding a Program:
Program expansion often follows an initially positive program evaluation. The program is expanded by introducing it into new schools if the effort is at a district level. Or, it can be expanded by adding more components to the program. For example, program evaluation can provide a good estimate of the staff support necessary for maximum impact of the program. This information is crucial for effective planning and resource allocation in new programs.  If the role of NTAs has proven to be a strength of successful conflict management efforts, strategic planning for new programs may first attend to shoring up the NTAs or encouraging their commitment to the program before implementation. In any expansion effort, whether in the same school or across schools, program evaluation provides the road map for success. It offers the opportunity to learn from previous mistakes, to preserve best practices, and to enhance the efficiency of the process.

Gaining Additional Resources:
People love a winner. This is an old maxim, but a true one when it comes to decisions about where to put already scarce resources. In education, there is little reason to throw good money after bad, especially when a lot of good programs vie for limited funding. Thus, one of the main reasons to do program evaluation is to make the strongest case for why you should continue to receive support for your efforts.            If you have solid program evaluation you can provide this evidence and prove that you have the competence to continue overseeing the processes and outcomes of the conflict management program.

Choosing Your Program Evaluation Focus         

“Research is formalized curiosity. It is poking and prying with a purpose.”
Zora Neale Hurston (1903 - 1960) US dramatist, author

Good program evaluation answers the questions “what happened?” ”how did it happen?” and “why did it happen”?  Program evaluation has two parts – process evaluation and outcome evaluation. Process evaluation is concerned with the process of implementing the program and whether that process is enhancing or inhibiting the success of the program. Outcome evaluation focuses on changes that have happened because of the program.

Process Evaluation
What are some of the processes that one might be interested in monitoring in conflict management? Although the processes may differ depending upon the program model being used, all programs will have some concerns in the following areas.

Assessment Process: The more elaborate the program model the more complex the assessment process should be.

Planning Process: Planning processes are critical to program success. Strategic planning insures that you are projecting difficulties and planning for them instead of being caught unaware.

Orientation Process: Once the assessment and planning is completed, the program needs to be introduced to the teachers, staff and students at the school. In most cases, some of those people have already been involved in the assessment and planning processes. Process evaluation can focus on how the orientation was accomplished.

Selection Process: This is the issue of who is chosen to directly participate in the conflict management program and what the impact of those participation decisions are for the program and school.

Training Process: All conflict management program models use some form of training.

Program Implementation Process: Up to this point, all activity has been preparatory. Now, the school has been oriented, the participants have been selected, the training has taken place, and the program is ready to be implemented. The process of implementation itself differs considerably by program model.

In peer mediation programs the program implementation process involves issues of program publicity, referral to mediation, utility of mediation, and linkage to other disciplinary activities. Process evaluation also can look at how the program is publicized and with what effect.

In curriculum integration models, process evaluation is especially important. It can explain how the curriculum was actually integrated in the classroom situation and the factors that influenced the integration process.

In whole-school models, especially those that have multiple components and involve external and internal groups, process evaluation allows emphasis on program development and coordination. In terms of development, process evaluation helps you explain the stages that the school went through in developing and implementing the various components. Difficulties or successes in one stage can be linked to the progress in another stage. Issues of coordination, especially when school programs involve a strong link to community, are particularly important to document.

Program Maintenance Process: If a program is long-lasting it will have to face the issue of turnover in personnel and participants. Schools that plan for the turnover in a manner that protects the program are able to weather the change. Unfortunately, many programs that make excellent process in all other areas were destroyed when turnover issues are poorly handled. Some questions that can be asked of program maintenance include the following:

Context/Environmental Factors: No matter how well a school plans for and tries to oversee its conflict management program, the reality is that the larger context or environment can influence what happens, sometimes beyond our control. When we talk about context or environment we recognize that there are multiple layers that could be considered. But for the purposes of process evaluation, we suggest attending to at least four.

The school context involves anything happening in the school that is outside the specific operation of the program. It could be the loss of an important leader or administrator that raises uncertainty about the future for the program.

The district context involves changes at the level of the school district and how that impacts school activity. A simple example is the case of a labor action at the district level that affects teacher availability and willingness to participate in an activity like conflict management.

The community context concerns the pressures and supports that are present in the surrounding neighborhood, community, or city in which the school is located. For example, a city may be undergoing cultural or racial tensions that increase incidents of violence and raise awareness for the need for constructive conflict resolution. In that environment, school programs may experience a resurgence of support and resource allocation.

Finally, at the larger societal level, there may be events that achieve statewide or national visibility that impact a school program. Although the influence may be indirect, it is nonetheless present. We all remember the terrible shooting at Columbine High School in Littleton, Colorado. Following that incident, there was a strong increase in utility of and support for conflict management efforts throughout the country. Similarly, when Attorney General Janet Reno proclaimed violence prevention and conflict management a critical need in our schools, her support became a rallying cry that educators found effective for arguing for program development and expansion.

Click here for samples of process evaluation forms that you can use or adapt for your CRE program process evaluation.

“True genius resides in the capacity for evaluation of uncertain, hazardous, and conflicting information.”
Winston Churchill (1874 - 1965) English statesman, author

Outcome Evaluation
Outcome evaluation focuses on whether and to what extent specific, tangible goals and objectives established for the program are achieved. The emphasis is on the outcomes of the program and your ability to document them. You have two basic goals in outcome evaluation:

(1) to document what happened in terms of utility or frequency (as opposed to how it happened in process evaluation) and

(2) to document what changed as a result of the program. In the first area of outcome evaluation, people attend to questions about the extent of involvement or activity. For example, how many students were actually trained? How many parents attended the workshop sessions? How many cases went to mediation? How many cases were resolved in mediation? How many times were student mediators linked with community mediators in community mediation situations? 

In the second area of outcome evaluation, the notion of proving change requires that you have some way to compare what the situation was before the program with what happened during and/or after the program. Most of us are familiar with pre-test and post-test designs and the use of control groups that are necessary for this kind of outcome evaluation.

The types of outcomes that can be evaluated are as varied as the types of programs that exist. However, we can talk about five general kinds of outcomes that most programs are interested in at some level.

Skills/Abilities Learned: As mentioned in Day 1, conflict management teaches foundational abilities necessary for the enactment of constructive conflict management. Thus, outcome evaluation may focus on questions about how well children learned the skills of active listening, perspective-taking, empathic response, generation of alternatives, anger control, etc. How well do mediators learn the actual process of mediation? How well do parents learn the process of emotional coaching and anger management? The key is to clearly specify the skills and abilities that you are trying to teach in the program and to make sure that these are being evaluated in terms of how well the program developed these skills or abilities in participants.

Attitudes Changed: Many conflict educators are interested in helping students adopt more pro-social attitudes. Depending on the emphasis on social justice concerns, the degree to which a social justice orientation is developed through exposure to the conflict management may be a critical outcome to be monitored. If the guiding motivation is violence prevention, the overriding outcome of concern may be attitudes toward violent and aggressive action or the tendency to make hostile statements about another that may escalate conflict. For whole school programs, an outcome of interest is usually school climate, or the attitudes that teachers and students have about the school. Evaluators are often asked to document how school climate changes over the course of the program implementation. 

Behaviors Changed: Given the link between conflict management and violence prevention efforts, an important outcome to be evaluated for many people is the degree to which violent behavior is decreased as a result of the program. How much less likely is a student to actually fight as a result of training? How many fewer suspensions or expulsions have occurred in the school since the program has been in effect? Have truancy rates dropped or increased? Are students using fewer disruptive behaviors in the classroom? Are teachers holding fewer disciplinary conferences with parents? Are students behaving more collaboratively in home or community settings?

Program Utility: Questions of program utility have to do with the extent to which conflict processes taught have actually been used in the school. In many programs this is the most common form of outcome evaluation. As mentioned above, this focus often assesses how frequently mediation is used, by whom, and with what outcome. Program utility issues are most germane to mediation programs or to the mediation component in peaceable classroom or peaceable school models.

Resources Created: Sometimes outcome evaluation can focus on resources that are created as a central or peripheral purpose of the conflict management program. These resources fall into three categories:

Click here for a worksheet that helps you consider your needs for evaluation and asks you to consider the kinds of evaluation that are most critical for your conflict management program.

 

Clarifying CRE Program Goals with Stakeholders

In Day 1 of this course, we introduced you to a list of goals for CRE programs. Here, we use an elaborated list of goals to help guide conversations with your stakeholders. These conversations are very important BEFORE you conduct an evaluation. If you are unclear about what you are trying to achieve, you’ll probably waste valuable time and money in an evaluation without a clear target.

As you review this list, begin thinking about which goals are clearly high priority for you and your school.

Create a Safe Learning Environment

Create a Constructive Learning Environment

Improve Classroom Management

Enhance Students’ Social and Emotional Development

Create a Constructive Conflict Community

CRE programs often have multiple goals. It is most helpful when you look at the program goals before implementing the program and the evaluation. If you are just beginning to plan your CRE program, the more time you spend clarifying your goals, the better off you’ll be. If you have already implemented a CRE program or are thinking of expanding your CRE program, considering program goals is still a very valuable step. And in all cases, knowing your goals becomes a blueprint for how and what you want to evaluate.

Click here for a worksheet to help you prioritize your program goals and build consensus around those goals with other stakeholders.

 

Assessing Resources

Now that you have an evaluation plan, you know what you want to find out.  Next, you need to take a realistic look at the resources you have to conduct the evaluation.

Information Resources: When you are planning your program evaluation, consider the extent to which you have available information about related research in the area. In many cases evaluators may spend time and energy re-inventing the wheel because they are not aware of other research or program evaluation that has been done.  For example, say your school wants to use a training curriculum that has not been used in the school before. Some stakeholders may have a strong interest in having program evaluation that assesses the students’ reactions to this curriculum. Before deciding to devote energy to this task, see whether other schools in your district, or other schools with similar student populations, have used this curriculum and reported on student satisfaction. In many cases, questions have already been answered, relieving you of the responsibility to repeat the effort. That will free you to spend resources on questions that are important and unanswered. In the last part of this Day 4 material we have a section on a review of recent CRE research to help you decide whether evidence of your program’s success may already exist and can be used to bolster your evaluation.

Data Availability: You should also find out whether data are available or can be made available to assess program impact. For example, if you are interested in seeing whether the conflict management program reduced violence significantly you will need to have baseline data of violence before the program begins. Do you know whether the school or district keeps this data on hand?

Commitment of Stakeholders: Now we are talking about the willingness of the stakeholders to participate in program evaluation processes. This may seem a given, but anyone who has done program evaluation can tell you it is one of the most difficult factors. The reality is that people get busy or do not provide you with the information they once promised--information that is critical to your program evaluation. Talk with stakeholders about the kinds of support they will give, the kinds of information they will provide and the consequences of not getting it. If stakeholders are reluctant initially, you can almost be certain they will not be cooperative when you need the information.

Program Evaluation Experts: Do you have access to program evaluation experts who can serve as mentors or guides during your initial program evaluation experiences? Often area universities or school districts have people on staff with this expertise.

Staffing: Do you have people in the school who are willing to act as program evaluators? This can be more time consuming than expected. Do you have people who can help prepare surveys, collect data, record data, read reports, analyze data, and write reports? The need for clerical kinds of support is greater in process evaluation than outcome evaluation. But the need for expert aid (for statistical analysis) is more pronounced in outcome evaluation.

Money: Do you have the money to pay for research support, office supplies, copying costs, etc.?

Time: Do you have the time to do the evaluation? Does the school staff have time to devote to it? Do teachers have time to allow data collection in their classes?

Technology: If you are doing statistical analyses, or are planning on using graphs in your report, do you have the necessary computer software?

In an effort to help CRE program evaluation and to provide some of the resources indicated above, the document “Looking for Success: Evaluating Your CRE Program” provides a detailed description of the six step process for CRE program evaluation. It also provides proven qualitative and quantitative measures for use in evaluating CRE program success. You can download this document for free from the following web site (http://disputeresolution.ohio.gov/schools/evaluatingcrep.htm).

Research Proven Benefits

Researched and Approved Programs

Reviews of Prevention Programs/Prevention Database: www.casel.org/sel_resources/fedrevlinks.php
Blueprints for Violence Prevention Office of Juvenile Justice and Delinquency Programs and Center for the study and Prevention of Violence: www.colorado.edu/cspv/blueprints
Safe Schools, Safe Students: A Guide to Violence Prevention Strategies. Drug Strategies. www.drugstrategies.org/pubs.html#safe
Safe and Sound: An Education Leader's Guide to Evidence-Based Social and Emotional Learning (SEL) Programs. www.casel.org/projects_products/safeandsound.php

The No Child Left Behind Federal Legislation stresses the need for educational programming to be research based.  Trying to choose a program and trainers may seem overwhelming, but there are many resources available to assist you.  In the last decade the research has shown that when effective strategies are used to teach staff and students the skills and concepts of classroom management, these skills can also be applied throughout the school community and can lead to reductions in discipline challenges, improved academic achievement, and a more positive overall school climate.  Below are several documents that have reviewed the research on various programs and include some of the programs that are approved for use with federally funded grants.

Attached is a research review article from Conflict Resolution Quarterly published in 2004 that summarizes relevant research in CRE. In this document there are also references to a variety of other literature review articles on specific programs (like Teaching Students to be Peacemakers) that may be of interest to you.

Click here to link to a .pdf file of “Conflict Resolution Education: The Field, the Findings, and the Future” by Tricia S. Jones, published in Conflict Resolution Quarterly, 2004).

If you are thinking of making a presentation and would like resource materials concerning current research, the following links will provide you with an annotated bibliography and an accompanying PowerPoint presentation summarizing CRE effectiveness in terms of CRE goals discussed in this course. A brief summary of the research in terms of goals is provided here:

Goals for CRE and Evidence of Effectiveness

 

Click here to access the annotated bibliography and matching PowerPoint presentation on “Proven Benefits of Conflict Resolution Education.”

 

Discussion Questions

Discussion Questions

Please think about the questions below and share your responses with colleagues.

How might program evaluation help you in starting or maintaining your CRE program?

What kinds of evidence seem most persuasive to Administrators? Parents? Funders? Other teachers?

What resources would be most helpful to you in planning and conducting your CRE program evaluation?