Return to Class Homepage

ELFH 606 Evaluation Article Critique

Winnie Cohron, Tony Niemann, Mark Schneider October 1, 2006

Howe, K. & Ashcraft, C. (2005). Deliberative democratic evaluation: Successes and

limitations of an evaluation of school choice. Teachers College Record, 107, 2275-

2298

Purpose

Howe and Ashcraft describe an evaluation conducted by the University of Colorado in Boulder regarding the Boulder Valley School District’s school choice policy. The purpose of the evaluation, authorized by the Boulder Valley School District, was to systematically examine the effects of open enrollment after five years of unexpected growth. Open enrollment meant the option of parents to choose a school within the Boulder Valley system based on available spaces for students. Open enrollment had been district policy since 1961, but with significant district growth during the 1990’s, and the availability of more types of schools from which parents could choose, the practice begged for a review.

Evaluation Plan / Results

A participatory program evaluation, deliberative democratic evaluation, guided the evaluative design and implementation of the commissioned yearlong Boulder Valley School District open enrollment (school choice) policy. The school district requested that the evaluation design should be objective and remain independent of the district, the school board, and another concurrent study being conducted by the district, the Colorado Association of School Boards and Kettering. This request necessitated an adjustment in thinking about the evaluation process. The evaluators promised that the deliberative democratic process would maintain objectivity in that there were no predetermined goals or conclusions, and in that significant issues would be reviewed through obtaining the views of those peripheral constituents whose opinions may not have previously been noted. Nonetheless, this promise made the involvement of all stakeholders less of a possibility.

Deliberative democratic evaluation grows out of active inclusion of all stakeholders who engage in elucidating dialogue (scrutinizing more than clarifying), in order to deliberate together until they reach consensus toward a workable conclusion. The school choice deliberative democratic evaluation plan was successful in promoting the principles of inclusion and dialogue, but was unable to fulfill the deliberative phase due to district restraints. Surprisingly, the deliberative phase occurred independently of the formal evaluation months after the study was complete due to the work of the press, which continued to spark public discussion through ongoing articles and editorials.

Evaluation data was collected through parent and educator surveys in all Boulder Valley schools; focus group discussions across a variety of school groups, school levels and school types; follow-up surveys of all principals in the school district; a random telephone survey of parents not actively involved in the school district; and statistical records of open-enrollment patterns, test scores, demographics, funding and fundraising. Surveys, focus groups, statistical data and the random telephone survey provided inclusion. Focus groups afforded opportunity for dialogue which was overwhelmingly more clarifying than scrutinizing.

Inclusion of all stakeholders, especially minority Latino families, and schools with a majority of lower socio-economic level families, provided equal voice to those in the district whose voices may typically have been unheard. Conducting focus groups at each school to discuss the impact of open enrollment on their school equitably called forth values, opinions, and needs from all areas of the Boulder Valley district. Written survey data from all parents and teachers allowed everyone to articulate their views. Attending to the specific voices of school principals permitted an up-close look at the views of those who carry primary responsibility for the success of each school in the district. Random phone surveys unlocked the door to a sampling of taxpayers in the community who may not have responded to any other method of contact. The evaluation methodology elicited input from all stakeholders in the district: parents, teachers, school principals, community members. Only the students were not involved in the data gathering. Viewed from the perspectives of all the participants who were involved in data gathering, the results of the evaluation should be valid (measuring what it says it measures) and reliable (findings would be consistent if the same process was conducted again).

Implications for Stakeholders

The findings and subsequent twelve recommendations published by the evaluators compelled the district to take a hard look at previous open enrollment practice. Some Boulder Valley school board members were not surprised by the evaluative results, while others were appalled that such inequities existed in their district.

District leadership acted quickly on the recommendation to centralize open enrollment procedures instead of allowing individual schools to continue to control them. The district improved the distribution of information about school choice to all parents, including the translation of materials into Spanish. Restrictions on how neighborhood schools (as opposed to district magnet schools) could recruit for open enrollment were lifted. The district conducted a study regarding cost of providing transportation to students who wanted to pursue open enrollment opportunities, but found that the cost of such transportation would be prohibitive.

A cap size was placed on the number of students who could enroll in district magnet schools. The district funding formula was adjusted to provide more money for schools with higher numbers of low-income students.

Each of these changes in district practice took place as a result of evaluation recommendations. Some of these decisions were more controversial than others. Nevertheless, the district chose to initiate changes in the order in which it felt would be most efficient and beneficial to students. The authors suggest that the most controversial of the recommendations, specifically, free transportation and redistributing fundraising dollars among schools, were not implemented, nor were they ignored. District leadership moved slowly forward as the result of press coverage which kept the deliberation portion of deliberative democratic evaluation active after the evaluation recommendations had been delivered. The evaluators felt that, while inclusion and dialogue had occurred during the evaluation process, the “findings and recommendations provided grist for subsequent deliberation [and] contributed significantly to public deliberation about Boulder Valley’s school choice policy and changes in it” (Howe & Ashcraft, 2005, p. 2286).

Evaluation Improvement

Use of the deliberative democratic approach for this evaluation was both a failure and a success. The approach failed in the sense that the evaluators were restricted from employing all three general principles of this approach: inclusion of all stakeholders; critical dialogue with all stakeholders, and open deliberation where minority views are fairly presented. Without using these three guiding principles in this evaluation, the argument can be made that the deliberative democratic approach was not used. The approach succeeded on the other hand, since significant results were achieved as a consequence of publicity after evaluation findings were published. The publicity stemmed from press attention, which included an “ad hominem attack” on the lead evaluator after the evaluation was completed (Howe & Ashcraft, 2005, p. 2291).

More time for the study and more money to support the research process would be required to render full implementation of the deliberative democratic evaluation process (Howe & Ashcraft, 2005). Although Howe and Ashcraft (2005) requested that the deliberative democratic evaluation approach, “become a central part of the deliberative process”, district officials declined this initiative, claiming that the evaluators would not remain objective if they used this approach. Both evaluators disagreed with this claim, but acquiesced to the district’s terms. Negotiations concluded with a sense of distrust, with evaluators wanting to focus on issues of stratification, skimming, and funding inequities, while district officials were hesitant to address these controversial issues head on. Concluding negotiations in a way satisfactory to both parties could have improved this evaluation process.

House and Howe, who espouse a methodological advocacy stance in leading evaluations based on the deliberative democracy approach, require what Robin Mellow refers to as “critical friendships” (Mello, 2005). Perhaps the evaluators could have fostered these critical friendships in order to reach a compromise in their negotiations with the local school district. Given the time, money, and methods required to conduct an evaluation of this size and scope, the deliberative democratic approach may be impractical to conduct without a close working relationship between employer (evaluated) and employee (evaluator).

References

Howe, K.R. & Ashcraft, C. (2005). Deliberative democratic evaluation: Successes and

limitations of an evaluation of school choice. Teachers College Record, 107, 2275-

2298.

Mello, R. A. (2005). Close up and personal: The effect of a research relationship on an

educational program evaluation. Teacher’s College Record, 107, 2351-2371.

Return to Class Homepage