ABSTRACT NO. 002-0275
What grading can tell about a course – and it’s exam
Track: Innovation in teaching
Thomas B. Christiansen
Center for Technology, Economics and Management
Technical University of Denmark
DTU – Building 421
DK-2800 Kongens Lyngby, Denmark
Phone: +45 4525 6120
Fax: +45 4588 4337
E-mail:
Second World Conference on POM and 15th Annual POM Conference, Cancun, Mexico, April 30 - May 3, 2004
Abstract
The purpose of this article is to illustrate how the data from grading students in a course can give valuable feedback to 1) the effectiveness and efficiency of the exam and 2) if there are aspects of the learning process that need attention. Before investigating three propositions, the article briefly describes a newly-developed course in Operations Management at the Technical University of Denmark. Since the introduction it has increased the number of students by more than 20% annually. The findings are that male students get significantly higher grades than females, and that a certain part of the exam is not sufficiently difficult. These findings suggest that improvements should be made.
Introduction
Most university courses include some kind of exam. The purposes for this can be multiple such as:
· giving a grade that tell others (for example future employers) about the student’s level of understanding of the content of a course (here it is assumed that there is a relationship between “understanding of content” and grade, which sometimes is not the case)
· encourage students to actually study the curriculum of the course (some students might find “enforce” a more rightful word than “encourage”)
· give the teacher some feedback to how well the students have learned during classes as well as during other activities taking part as part of the course (this might work best when the teacher does not make the exam him-/herself)
This article go beyond the third bullet and uses the grading to evaluate the exam in itself, in order to see if it is effective as well as efficient.
At the Technical University of Denmark (abbreviated DTU) it has for ages been a tradition to finalize a course with a four hour written exam at a designated date in a two week “exam period” at the end of the fall or spring semester. These exams tend to have some similarities from year to year, having the same structure with about the same assignment problems with slightly changed numbers. In all courses, it is the teacher’s obligation to formulate the exam and afterwards correct the exams if he/she cannot find a suitable and willing ph.d.-student to do the job and finally give the grades. Most teachers find this task of making and evaluating exams a necessary, but not loved activity in their jobs.
Given that teachers have to spend some time on this task, it might be worthwhile spending some time evaluating the exam in it self and ask if he/she spends the time in a good way. Actually, what should be done is to study whether the exam is effective and efficient from the teacher’s point of view:
· Is it effective? (Do the different parts of the exam make the students behave the way that is expected from the teacher? For example, the exam might be structured so that it starts with a simple problem that most students can handle and at the end a complex problem is presented, that only the best and brightest students can solve)
· Is it efficient? (Is the time the teacher spends well used? If the exam for example both includes a multiple choice exam and a case study, it must be worthwhile for the teacher to spend considerably more time correcting the answer to the case study)
The article look at these questions by analyzing data from the exam of a specific course held in fall 2002 at DTU. The article is structured as follows. First the course is shortly described. Then the next section describes the exam and what the teachers wants out of it (“performance criteria”), and at the end of this section three propositions are suggested. The next sections describe the data behind the analysis, the next again analyze data and presents the results. Finally a conclusion is presented as well as suggestions for improvement.
The course
The course 42405 Introduction to Operations Management was given for the first time in fall 2000. It replaced a previous course Introduction to Industrial Management, which has been held in the same format and content (including textbook) for some 10 to 15 years. In 1999 only 39 students assigned up for the exam. This is down from about 100 when this author followed the course in 1992. This decline was obviously a problem, and even more problematic was the fact that it was an introduction course after which students could take the more advanced courses at the department. Therefore, it could be foreseen that also the advanced courses’ attendance would decline in the future as well.
As part of this authors duties during a ph.d.-study, he was asked to “revitalize” the course. A precondition was that one of the two other teachers from the previous course also should take part in it and have about half the sessions on the new course leaving the other half to the new teacher on the course. However, it was the new teacher that suggested content and structure for the course, which more or less was approved by the older teacher.
The goal for the new course was – besides that students learn the content – to attract good students from all departments of DTU and show then how exciting Operations Management is so that they will continue taking more advanced courses afterwards at the department.
As background information, the Department’s name is Department for Manufacturing Engineering and Management, and is one of 14 departments at DTU. Students are taking courses from multiple departments; hence there are no guarantees that students will follow courses at a certain department. Of course there are a number of outlined lines of study that students can follow, which implies following certain courses, but still things can change quire rapidly as the word spread about good and bad courses.
This course is normally taken at the third or fifth semester of a 10-semester candidate degree and is equal to a Operations Management course for an US bachelor degree.
When developing the course a number of issues were raised:
· Which textbook to choose
· Involvement of a case company
· Group work with diverse student profiles
· Classes should be interesting and varied with regards to structure and learning process
· Keep a written exam that test the learning of students (will be covered in the next section)
Textbook
It was chosen to select a main stream Operations Management textbook. The textbook that was used on the previous course was written by the former teachers, and then not updated for many years. In order to avoid problems with finding time for updating, it was chosen to find a textbook that likely would be updated quite often. An American textbook was then chosen.
Case company
Given the choice of an American textbook there were some concerns that the “reference company” in the textbook would be rather large compared to the average Danish company. Hence it was debated to involve a Danish manufacturing company to contrast the examples in the textbook. A well-known and respected Danish manufacturing company situated just 10 kilometres from DTU was willing to participate. Participation involved guest lectures, a visit early in the course and later hosting students groups that had a small project focusing on an Operations problem in the company.
Student groups
Assigning students to groups can be a challenge. Here it was chosen that the teachers (the author) did the assignment for them instead of letting him or herself get into groups with friends. Here, the goal was to assign groups in which nobody knew each other on before hand.
Interesting and varied classes
Classes were 3-hour sessions, which implied that the normal lecture format would not work if the attention of students were to be kept. Other activities were included, such as case studies that were discussed in groups, problems that were solved in groups, guest lectures and different kinds of games.
In total 16 sessions out of 26 were “normal” sessions with lectures and then something else as outlined above. The last 10 classes were spend on visiting the case company, making the report of the project, playing a simulation game and finally one of the sessions were actually spend on the written exam. This exam was held just after the last chapter of the textbook was covered and before the students should work on the project in the case company. We believed that this would be a good way to get the exam out of the way so the students could focus on the case company. Two other reasons for placing the exam at this time was that it would encourage students to read the textbook through the semester instead of just before exam, and the other – and much more pragmatic reason – was that the teacher correcting the exams would not have to do this over the Christmas holiday.
To summarise, the course is in no way extraordinary, a textbook is taught, but besides this several activities were carried out in order to illustrate the content and create a more exiting learning environment. Based on the number of students following the course it seems to have been a success, since it has increased about 20% each year since the start in 2000.
The exam
It is the written exam from fall 2002 that provide data for the analysis. The exam lasted for four hours and it had three parts, which aimed at testing three levels of learning, which in this case also meant that the level of difficulty was increasing from one part to the next - or at least that was the plan.
The students were told the following about the three parts of the exam (translated as closely as possible from Danish):
Part 1: Multiple choice questions (weight 25%, equal to 1 hour of work)
Purpose: to evaluate basic knowledge
Content: Part 1 consist of 40 questions with four possible answers
For each question there is only one correct answer
Part 2: Problems and short answers (weight 50%, equal to 2 hours of work)
Purpose: to evaluate understanding and skills in applying/combining content from course
Content: Part 2 consists of a number of questions that can be either problems to solve using formulas or short answers of clarification
Part 3: A case (weight 25%, equal to 1 hour of work)
Purpose: to evaluate application and capability of combining content from course in relation to a specific problem as presented by a case
Content: Part 3 consists of a case description and a number of questions
The purpose of Part 1 is basically to ensure that students have read the textbook. The purpose of Part 2 is then to evaluate the students’ abilities to work with problems, i.e. use formulas to calculate for example the order size using EOQ-formula. This part “tests” some of the group work during the course. Actually “testing” means that by having this part in the exam, students are encouraged to participate in this group work. Finally, Part 3 has the purpose of evaluating the students’ capabilities in working with a case, not solving it as such.
The underlying assumption is that it takes more skills (here interpreted as learning) to be able to answer Part 3 than Part 2, as well as Part 2 is supposed to be more difficult than Part 1. Actually, Part 1 and 2 are to some extent about different levels of “reproducing facts”.
As mentioned before, the course is an introduction course, hence from the Department point of view there is no interest in failing more students than necessary. This means that students that are capable of providing a good performance on Part 1 and Part 2 should be able to pass the exam. In order to be able to reward the better students, we have included a somewhat more difficult Part 3 so that the grades can be distributed along the scale.
Until now, issues of effectiveness have been described. The issue of efficiency is addressed now. An administrative assistant corrects Part 1, while the author corrects Part 2 and 3. The author and the other teacher on the course give the final grade. The notion of efficiency in this context refer to the fact that if the multiple choice alone can make sure that the grades are spread across the students so that the good students get good grades and so on, there is no need to spend a teachers time on correcting exams. Especially it is not efficient if part 3, which is the most difficult and time consuming to correct not contributes to the final grade. One way to investigate this matter is looking at the first proposition:
Proposition 1: There is a significant difference in the average point score between the three parts.
The next thing to look into is differences between students that have failed and take the course again (or more correctly: take the exam again) (called repeaters) and new students (called newcomers). It is here assumed that the reason for students to fail either is lower intelligence or lack of preparation during the course and before exam. When phrasing the following proposition it is further assumed that this will repeat, hence:
Proposition 2: Repeaters get a significantly lower grade than newcomers.
Finally, male students dominate DTU in numbers, only about 25 – 30% of students are female. Therefore, it might be interesting to investigate if there is a difference in grading between male and female students. Actually, it is a goal for the course to have at least the same share of female students at this course than at DTU in general. In 2000 only about 5% were females now in 2002 it is about 30%. Therefore, it would not be good if, i.e., females in general get lower grades than males. However, there is no reason to assume that is the case, hence: