ETAD, Learning, Math, Professional Development

Where We Are and Where To Go Next: Creating a Survey

My learning journey in EDCUR 809 has taken me to assignment 5…creating a survey. I’ll be honest my first impressions of this assignment were Easy, How hard can a survey be? Create a few questions, get back some data, make a nice Excel bar graph and Voilà ..survey and analysis done.
Well I was wrong. Really wrong.

I reflected to a colleague that this truly was an “end of ignorance” experience. Now that I know more it was not easy. Every question caused me to think, wonder, and reflect on the question “Will this provide us with the information we need?”

I began by deciding to create a survey for the PreKindergarten to Grade 5 Mathematics Learning Community. I wanted to create a tool to assess the degree to which the math learning community members have reach the medium term (year 2)outcomes and provide recommendations for next year. I wanted a survey that would shed light on where we are and where we are going as a professional learning community. My first step was to revisit the outcomes through the lens of assessment. The outcomes for year 2 are:

  • increase understanding of mathematics content, pedagogy, and instruction
  • increased enjoyment of teaching mathematics
  • increased understanding of content development across mathematics curricula (grades prior or succeeding current teaching assignment)

To strengthen the survey assessment and analysis I also wanted to consider Guskey’s (2000) five levels of professional development. These levels are used by my system’s Staff Development Team to evaluate the effectiveness of our professional development. The levels are as follows:

  1. Participants’ Reaction
  2. Participants’ Learning
  3. Organization support and change (are barriers being reduced)
  4. Participants’ use of the new knowledge and skills
  5. Student learning outcomes

Specifically, I felt that my survey would focus on levels 1, 2, and 4. (However, based on feedback I revised my survey to provide an opportunity to gain insight into number 5 as well)

With both my outcomes and the five levels of professional development in mind I created the initial survey. From there I shared it with colleagues. They provided me with feedback that lead me to revisions. Next, I piloted the survey with 4 members of the Mathematics Learning Community to get their feedback. This lead to more revisions, more pondering about what data this survey would provide and questions about if the data would be adequate to shed light on the my inquiry into effectiveness and actualization of outcomes.

Initial Version of the Survey

Initial Math Learning Community Survey

Revised Version of the Survey

Revised Version of the Math Learning Community Survey

(please note that this survey will be put into an online tool. The open question text boxes provided in the document are place holders and do not reflect the amount of space participants will be given to complete their responses.)

This survey will be put into an electronic form to be sent out to all community members around the middle of May. From there the plan is to bring together a small group of members to help with the analysis of the survey, provide additional insight into the actualization of outcomes and make recommendations for next year. I hope that the survey and the follow-up analysis will provide the information we need to move forward effectively.

Image: Objects In The Rearview Mirror by KrissyVenosdale- http://www.flickr.com/photos/venosdale/6876773480

ETAD

Choosing My Own Evaluation Adventure- Developing a Plan

Assignment number 2 of my Program Evaluation class has me digging into a given program and choosing a model that I feel is appropriate to evaluate the program.  The program we have been provided with is prenatal exercise program for Aboriginal women.  This program was designed to address the concerns around the high incidence of Gestational Diabetes and Type 2 Diabetes among North American’s Aboriginal population.  This Saskatoon based program was initiated around an inquiry into whether physical activity among Aboriginal women during their childbearing years could play a major role in optimizing healthy pregnancies and in reducing rates of type 2 diabetes in future generations.

CIPP Model
Adapted from Stufflebeam’s CIPP Evaluation Model (1983)

Once again I would suggest that the CIPP Model by Stufflebeam would be an appropriate model for conducting an evaluation on this program.  My recommendation is based partially on the belief that this model would be a good fit and partially because it is one the model we have studied that makes the most sense to me…just being honest.  Stufflebeam’s model is straightforward and focuses evaluators on four core concepts- Context (Goals), Input (Plans), Process (Actions), and Product (Outcomes).  By applying these four concepts to the Aboriginal Prenatal Exercise Program I believe program designers would be able to gain a valuable insight into effectiveness of the program to date and necessary information into how to improve.  In this way the evaluation would be formative in nature.

I believe this model will work in this case for several reasons:

  • Evaluation within each concept will provide information into the effectiveness/ quality of the program goals, plans, actions and outcomes.  This would provide insight into what works and where potential changes need to be made.  
  • Data from each component could be gathered and evaluated.
    • Goals– Program Goals around decreasing Gestational Diabetes among Aboriginal women could be evaluated in comparison to research and similar studies across North America.   Questions around Is the research sound?  Are the connections made valid? Is this inquiry worthy of the time/ resources needed to actualize? could be investigated.
    • Plans– In evaluating the plans laid out by the program evaluators could look for coherence between the objectives of the program and each of the programs components.   The program appears to components around exercise, nutrition, pregnancy, education, and creation of a supportive community seem to be in place.
    • Actions–   Many actions were taken in this program to actualize the plans put in place.  Data from each action taken gathered.  In this component I would look at collecting data from the numerous stakeholders in the program.  In particular data from the program participants would be invaluable.  Questions around Did this program meet your needs?  What did you learn from the program?  What parts of the program were most valuable to you?  What changes would you make? What factors led you to attend or not attend? could be investigated through interviews, surveys, and/or  focus groups.
    • Outcomes– This part of the evaluation would led one to look into the medical data of participants, their children, and non participants as it exists now and into the future.  By looking into this data one could see if the goal of reducing the incidence of Diabetes during pregnancy  into the future and among the offspring of the women is being met.  By studying the health stats of non participants as well, evaluators can see how the results of non participants compare to those of participants.  Much of the data needed to make a summative evaluation would not be available this early into the program however current health data could be used for formative evaluation.
  • Each component would allow for multiple sources of data to be collected and analyzed   In this way the evaluation could be very thorough and comprehensive.  Input from the many stakeholders, and their perspectives should be used to inform this evaluation.  It would be very important to consult with not only the medical personal involved in this program but also the community members whom this program effects.  In particular attention should be given to gather and honor the perspectives of Aboriginal Elders and Knowledge Keepers within each component of the model.

I am sure that if I was to actually conduct this evaluation more questions would emerge and the plan for evaluation would evolve to become more complete and comprehensive.  This assignment has created an opportunity for me to continue my learning with Program Evaluation.  One thing I have learned is that the knowledge of the content is not a prerequisite for beginning an evaluation.  I know very little about diabetes yet by applying the limited information I was given to the CIPP model I was able to “see” a plan for evaluation.  This makes me wonder if a richer knowledge to the content would have led me to a different evaluation plan.