Reading Response: Chapter 1 & 2

EC&I 809: Program Evaluation

May 11, 2009

Kim E. Engel

Part 1: SQ3R

My first response to Chapter 1 was “Oh, my goodness – this is overwhelming!” … but as I got into the material, I found it was more approachable than I first thought. I have had a class that addressed evaluation and how it differs from assessment, so this reacquainted me with the terminology and meanings. The authors made the connection between evaluation and ‘defensible criteria’ early on in the first chapter. They also drew a very clear line between research and evaluation and researchers and evaluators. The distinction in the purpose of each stood out for me – research seeks conclusions and evaluation leads to judgments.

I felt the examples that were given added meaning, they illustrated the definitions and charts that were used. I appreciated how the authors delved into the purpose for evaluating, in fact I found six ideas explored relating to the purpose of evaluation. I felt the purpose that most accurately applied to the school situation was to: render judgment on the worth of a program, assist decision makers responsible for deciding policy, and serve public function (Talmage). In my opinion, this dealt with the guidance and judgment that surrounds evaluation. Judgment was further highlighted by listing the ‘influences of judgment’ discussed in Chapter one – experience, instinct, generalization and reasoning – pointing out that any or all of these can cause a sound or faulty judgment. It was pointed out that judgment is an informal evaluation and there problems in using informal judgments, essentially a 50/50 chance of success, because: they can result in faulty or wise judgments, and, they can lack breadth and depth because they lack systematic procedures and formally collected evidence. Evaluators need opportunities for observation and a strong history of experience in order for judgments to have merit. The lack of opportunity to observe and lack of experience inform and bias judgment. The authors point out that in spite of this, sometimes informal evaluation is the only practical approach.

I was surprised how exhaustive the definitions where for formative and summative evaluation – I have never had more than a paragraph on either. The authors point out that formative evaluation is often found in the early stages of evaluation, whereas summative occurs in the later stages. The quote that summed up the difference between the two forms of evaluation is, “When the cook tastes the soup, that’s formative evaluation, when the guest tastes it, that’s summative evaluation.” (p. 17)

I appreciated the list of activities a professional evaluator is often responsible for – it certainly sounds like the administrative side of my job! I am often found negotiating with stakeholders; developing contracts, not hiring – but interviewing and overseeing staff, managing budgets, identifying the disenfranchised, working with advisory groups, collecting/ analyzing and interpreting data, communicating with stakeholders, writing reports, considering ways to disseminate information, meeting with groups to report results and recruiting others to evaluate … the professional evaluator and I have a lot in common!

The opening questions of Chapter 2 helped me get my bearings for what was to come in this chapter. In fact, before I began looking the chapter over, I simply sat and thought about the questions. I appreciated the link that was made to historical events and how they influence, even our beliefs around evaluation. My moment of disbelief was on p. 31 when the authors reveal that in Boston (1845 and 1846) delivery of printed tests in several subjects was a first in attempted objectivity, “These two developments in Massachusetts were the first attempts at objectivity measuring student achievement to assess the quality of a large school system.” The descriptions of how evaluation was conducted relating to accreditation and the ‘walk-through observations’ that were used to determine whether universities and secondary schools could be accredited created a link for me to chapter ones description of summative and formative evaluation – use of observation and ones own experience to pass a judgment.

The description of program evaluation (1940-64) highlighted the role researchers and social scientists had on what was evaluated in this era. “As in the past, such studies often focused on particular facets of the program in which the researchers happened to be interested.” (p. 34) But the authors point out that social scientists “began to focus their studies more directly on entire programs” (p. 34) This brought to mind how we can be selective in what we choose to focus on in program evaluation instead of looking at the whole we can get distracted by parts of. I am teaching one subject to all the grades in my school. I work diligently at communicating my observations and assessment results with the teachers in our school. The goal of such communication is for the teaching staff to work as one in meeting student needs and advancing their progress. If I stayed in my room and kept everything to myself, the potential growth and assistance for students would be hindered. I think that is what is being described by the historical reference to this era.

The era that really gave the evaluation strand a boost had to be 1973-1984 when evaluation became a profession. The introduction of training students to become evaluators with professional association created a separate entity and diversity with a developing expertise in the field. A real shift occurred at this time, “Evaluation moved beyond simply measuring whether objectives were attained, as evaluators began to consider information needs of managers and unintended outcomes. Values and standards were emphasized, and the importance of making judgments about merit and worth became apparent.” (p. 39) The debate and controversy these ideas brought forth is mentioned, but more importantly the ideas of goal-free evaluation is introduced, as well as the notion of examining process and context. This brings memories back to me as a 1983-87 education student and emerging teacher. The emphasis was very much on process and ‘how we get to where we are going’.

The era we now find ourselves in is really a hodge-podge of ideas and tools. I do not see or feel forces of ‘this is the right way’ like I did early in my career. I still observe the band-wagons as they drive by, but I do not see a consistent loud voice of endorsement of ‘one way’. The one concept relating to evaluation that I do see as wide-spread, through educational institutions, government and business is ‘outcomes’ and the link they have to bottom-line thinking – what do we want children to be able to do in the end? Just as governments want to know, did the program we were running do what we wanted it to? Did it do better than we hoped? Did it do other things we had not anticipated? … I was introduced to outcomes in my work with the federal government in the early 2000s. They were clear to point out that objectives were no longer acceptable, as a teacher I was shocked. My undergrad and teaching years were all about setting objectives and deciding how to evaluate if they had been met. Through that training and practice, I was on a journey with my students and I decided if we had reached our destination. The outcomes approach was very different, we were told where to go and it did not matter how we got there. Now we are told the destination and how to get there. Prescription is now the norm. I think they are good parts and bad parts of this, perhaps uniformity was necessary with some tightening up of what we are doing. At the same time, not all students are the same so comparison can stifle there unnoticed achievement.

Part 2:

What does this mean to me as I fulfill my various roles?

Essentially it provides an awareness of the many facets of evaluation and the appropriateness of different types in different situations. For example, to evaluate school programs several types of evaluation could be utilized: surveys; interviews; analysis of school data … a variety of forms demonstrates different sides of what is being evaluated. On a broader level, it reminds me that the evaluation programs I am experiencing will continue to evolve and change, which is all apart of growth. This forces me to open my mind further, I know I would dearly love “the correct evaluation methods” to be discovered, but there is not one way, or few ways – there are many. I think the connection to history is important for me to remember. In our age of ‘instant-everything-technology-obsessed’ society, the desire to have evaluators and evaluation tools to improve our practice and the results of our programs is strong.

How can I use this info to address a problem or an issue in my school, workplace or elsewhere?

I would look at what needed to be evaluated and then try to choose several forms of evaluation that would fit. I guess one of the biggest messages I got in these readings is there is more than one way to evaluate an issue – there is not one ‘made to measure’ evaluation that will fit any situation. Although, this not always acknowledged by those requesting evaluation. At present, data is collected on our students relating to an outcome on a test – one test. These numbers than are compared to benchmarks and other students in the division. In my opinion, the results are questionable. If more than one form of evaluation was used, a balanced picture might be more likely. This does fly in the face of ‘performance measurement and standards-based education’ where interpretations are made based on the evaluations, which does not necessarily fit the reality.

How does this advance my abilities as a participant in program evaluation?

I am more informed in how evaluation can be conducted, therefore I feel I would be an informed participant. I realize I have been apart of many evaluations without being cognizant of it – telephone surveys, mail questionnaires, product trials, focus groups … I think an important fact I have learned is that because evaluation methods are ever-changing and reflect society at the time, that there is always a hope of improvement and increased accuracy. This also means that I can play a role in change. We still live in a time where dialogue can occur in a democratic manner, although I am aware this is not the case everywhere. There are times when I feel my school division is not open to the possibility of maintaining a process view as it focuses solely on outcomes, but the historical portion of Chapter 2 does make me realize that being sold on a viewpoint of evaluation is all apart of the evolution of a viewpoint. Do I feel limited as a participant in program evaluation at present? Yes, actually quite restricted as an approved menu is all that is acceptable at this time. I do think this will change, perhaps not in my remaining career, but it will change as new considerations are taken into account and recognized as valid.