My graduate studies journey continues with my current class EDCUR 809. This class will provide me with the opportunity to examine models for the evaluation of educational and other programs. I chose this class because I wanted to strengthen my understanding of how to objectively evaluate the educational programs I deliver thus providing me with the insight and information necessary to make improvements. My first assignment has me looking at a completed evaluation, determine the model used and identify what I see as the strengths and weaknesses of the evaluation. So here goes…
The evaluation I chose to dig into was completed for the Literacy and Numeracy Secretariat of Ontario titled The Impact of the Literacy and Numeracy Secretariat: Changes in Ontario’s Education System. The mandate of the Literacy and Numeracy Secretariat is to actualize the provincial goal to improve students reading, writing, and mathematics skills through the building of instructional and leadership capacity of at all levels. This evaluation focused on reviewing of the actions taken by the Secretariat and determining if the education system, the educators and the students have benefited from those actions.
The model of evaluation used appears to be the CIPP model by Stufflebeam. In this model evaluators look into the following areas- Context (goals) , Input (plan), Process (Actions) and Product (Outcomes). I believe that the evaluation team focused on the input, process and products extensively. Specifically the objectives of this review in comparison to the CIPP model were to:
- Identify and evaluate the LNS initiatives input and process
- Determine whether and in what ways Ontario’s education system has changed as a result of these initiatives product/ outcomes
- Determine the extent to which these changes have benefited students and educator – product/outcomes
- Draw lessons from these findings, as a guide to the ongoing improvement of Ontario’s educational system – summative outcomes to inform decision-making moving forward. (p. 10)
In order to accomplish these objectives the review team took on the momentous task of reviewing literature, meeting with focus groups, conducting interviews and surveys, looking at student data, meeting with sample schools and numerous share holders in the education system. At the end of this extensive process the evaluation team concluded that Ontario’s Literacy and Numeracy Secretariat has had a major, and primarily highly positive, impact on Ontario’s education system.
There are many strengths with this evaluation. First of all it is very though. The input of numerous stakeholders (students, parents, teachers, administrators, trustees and the community) were collected, analysed and used to make recommendations. The sources of data are varied and include everything from conversations with focus groups, to student data, to surveys with teachers, and administrators, to the careful studying of products created by the Secretariat. Secondly, the evaluation was based on sound research in the areas of literacy, and numeracy as well as student learning. Lastly, the evaluations was completed by an evaluation team that was led by five researchers from the Canadian Language and Literacy Research Network based out of four Ontario Universities. I believe that a major strength of this evaluation is the diverse expertise that the evaluation team brought to the table.
One weakness I see in the report is the seemingly contradictory findings. The evaluation team recommends accelerating and intensifying the efforts to improve instructional and assessment capacity in numeracy, and continuing the initiatives currently in place. However, they also indicated that teachers are feeling overwhelmed with the pace of the information coming out and need more time to internalize the learning and apply it in the classroom. No indication is given as to how to do both. As well, the recommendation of increased emphasis on numeracy is not supported with any recommendations as to how to do this.