Wisconsin Teaching Fellow Final Report – John Colton, UW-La Crosse
1.Explain the question, problem, or issue related to teaching and learning you investigated.
In Physics, as in other areas, exams can be given in many different formats, or “exam styles”:
- students sometimes are prohibited from using any notes, while taking the exam
- students are sometimes given key formulas on the exam by the professor
- students are sometimes allowed to prepare a sheet of notes which they may use on the exam
- students are sometimes allowed to use their class notes and/or the textbook
- students are sometimes given take-home exams
The most common method that I personally have used, is to allow the students to prepare a one-page “note-sheet” for themselves, typically containing key formulas, which they may use for assistance during the exam. This is also perhaps the most common method of giving exams among other physics professors I have asked. The general feeling is that this will allow students to focus on “what is really important”, rather than “wasting time” memorizing formulas. The thought is that because the students see the fundamental concepts and equations so many times and places during their undergraduate career, they will end up having the most important things memorized even without having needed to memorize them for individual exams along the way.
However, as I was interviewing students after they took the ETS “Major Field Test” given by our department for the first time in 2006 (completely closed book & notes), I found that many of the students had not known some very basic physics equations. Things that I considered fundamental definitions and concepts—which must be surely known by every physicist—were viewed by some of the students as superfluous items which they couldn’t answer because they “didn’t have the formulas memorized”.
The broad issue I wished to explore, set up by this experience, is: are we doing more harm than good by not requiring the students to memorize equations and take exams closed-notes? More specifically, in this research project I wanted to get at issues such as the effect of exam style on students’ study habits and attitudes.
2.Explain the methods or approach you used to investigate the question, problem or issue.
During the Fall 2006 semester, in an PHY 335 (Electronics), I gave various styles of exams throughout the semester (one with a note-sheet permitted, one closed book, one open book/notes, and one take-home). At the end of the semester I interviewed the students in focus groups to see what they thought about the various exam styles. These focus groups indicated that exam style was perceived by students as having various effects. Some effects mentioned by multiple students included:
- accuracy of assessment of students’ ability
- stress level
- amount of learning that occurred
- amount of studying that occurred
- type of studying that occurred
- retention of material
Analysis of the focus groups’ comments allowed me to write a survey to be given the following semester, which would probe at some of the most important potential effects of exam style on students—see the attached document, “coltonsurvey.doc”
During the Spring 2007 semester, in PHY334 (Circuits), I conducted a controlled experiment: I split the class randomly into two groups. Group 1 took the first exam closed-notes, then the second exam with a note-sheet allowed. Group 2 took the first exam with a note-sheet, and the second exam closed-notes. (I.e. eachstudenttook one closed-note exam and one note-sheet exam). The experiment was performed in this fashion to eliminate apparent differences that could arise from e.g. the second exam being more difficult than the first exam.
After each exam, both groups were surveyed to find out how they had studied, how they felt they had done on the exam (and why), their anxiety level prior to the exam, anticipated retention of the material, and attentiveness during lectures. Differences between closed-notes and note-sheet groups were analyzed via chi-square tests to see which if any survey responses were statistically different between the two groups. Additionally, students were allowed to make open-ended comments on their experiences and opinions.
The groups’ exam scores were also analyzed to determine whether the note-sheet group did better than the closed-notes group, or vice versa.
3.Summarize the results even if they are preliminary.
The results were quite interesting—both to see where differences existed between the two groups, and where they did not.
Areas where exam style caused a clear difference
Study time: There was a very significant difference in time spent studying for the exams: 77% of the students said that they studied more for the closed notes exam than they would have studied for a typical exam. When asked how much they had studied, 60% said that they had spent more than 4 hours studying by themselves for the closed note exam; only 45% said that they had spent more than 4 hours studying by themselves for the open notes exam.
Exam study habits: There was a very significant difference in how likely students were to prepare formula sheets to help them study for the exams: 74% of the students did not prepare a formula sheet to help study for their closed-note exam. By contrast, all but one student prepared a formula sheet to help study (or to help use during) their note-sheet exam.
Areas where exam style may have caused a difference
Homework time: The amount of time spent doing homework may have been significantly different: students spent more time doing homework prior to their closed-note exam. Specifically, 37% of the students spent 4+ hours per week doing homework by themselves leading up to the closed note exam; by contrast, only 14% of the students studied 4+ hours per week by themselves leading up to the open note exam.
The chi-square analysis of this question yielded a p-value of 0.27. This indicates that there was likely some actualeffect here, because there was only a 27% chance that the differences in responses could have been due to chance alone. However, this falls short of the traditionalstatistical standard (of p = 0.05) to indicate a difference beyond doubt. Thus this is an area where further investigation is warranted.
Attention in class: The amount of attention paid in class may have been significantly different: students indicated they paid more attention in class prior to their closed-note exam. Specifically, 74% of students said that they paid a “large amount of attention”in class prior to the closed-note exam; only 59% of students reported paying a large amount of attention prior to the open-note exam.
Chi-square analysis of this question yielded a p-value of 0.18; that is, there is only an 18% chance that the difference in responses could have been due to chance alone. As with the “homework time” question, this is enough to indicate a likely effect which warrants further investigation, but not enough to state conclusively that the effect could not have been due to chance.
Areas where exam style caused nodifference
Exam score: Exam style turned out to not significantly affect the students’ exam scores: almost exactly half the class did better on their closed-note exam, and half did better on their open-note exam (relative to the class average). On the first exam, the note-sheet group scored highest; on the second exam the closed-notes group scored highest. That is, the same physical group of students scored highest on each exam. This contradicts the students’ impressions that they will score higher on note-sheet exams (see student comments below).
Perceptions of validity of exam score: After each exam style, the vast majority of the students (75%) felt that their exam score did not accurately reflect their ability, and was lower than they deserved. There was no significant difference between the two exam styles, however—students after the closed-notes exam felt like the exam did not adequately assess their true ability in nearly the exact same way that students after the note-sheet exam did.
However, a large difference occurred in why the students felt like their exam score was invalid:89% of the students in the closed-notes group listed absence of note sheet as a factor for why their grade was lower than they deserved, with 52% of those indicating that it was a moderately significant factor or larger. Since the overall level of complaint was no different than in the note-sheet exam, however, it seems that the lack of note-sheet had no real effect, but merely served as a focal point for the students’ complaints. (Students in the open note group attributed the inadequacy of the exam’s assessment to other sources.)
Anxiety: Although 48% of students felt like anxiety had lowered their exam score, this was nearly identical for students talking about the closed-note exam vs. the note-sheet exam. Thus students did not seem to feel that the “no note-sheet” contribution towards their anxiety level affected them beyond their typical anxiety.
Exam study habits: All of the students worked sample exam problems to help them prepare for both exams. 77% of students re-worked homework problems to prepare for exams, regardless of exam style. 98% of students reviewed class notes to prepare for exams, regardless of exam style.
27% of students did not read the textbook to help prepare for the exams, regardlessof exam style. 62% of students did nottake any notes from the textbook to prepare for the exams, also not depending on exam style.
Thus, the only study preparation item which was affected by exam style was the preparation of the note-sheet itself, as mentioned above.
Estimated Retention: Students were asked to estimate how much material they would remember three months from the exam. They did not answer differently for the closed-note exam as compared to the note-sheet exam: in each case, 25% said they would remember the material “slightly”, 62% said they would remember it “moderately”, 10% said they would remember a “large amount”, and 3% said they would remember “nearly all”.
Therefore, students apparently did not feel that they themselves would remember any better for the closed-note exam than they did for the open-note exam. This contradicts what several students answered in the open-ended question regarding advantages to a closed-note exam, where they said a closed book exam ought to help students remember the material better.
Student comments in open–ended responses
Also quite illuminating were many of the students’ comments in the open-ended response section. Some interesting and/or representative comments are given, sorted by common themes:
Benefit of a “closed-note” exam
Causes students to study more
•“[offers] a larger incentive to spend more time studying”
•“You are forced to study even more.”
•“…you don't have a false sense of security, like you do with a note sheet.”
Aids learning
•“You work harder and remember more”
•“[puts things in] long term storage”
Better assessment of student learning
•“more accurately reflects student knowledge/learning - material must be known better”
•“Making sure students really understand, and aren't just winging it based on a notes sheet.”
•“you rely on your knowledge and your understanding rather than just a note sheet”
No benefitto closed-book exams (many)
•“Memorizing formulas you can easily look up if needed is completely worthless”
•“No real point, formulas are typically forgotten soon after the end of the semester anyone.”
• “…my personal belief is that if an equation is important/used enough that it should be memorized, you will either eventually just remember it from looking it up so much, or get sick of looking it up and spend the time to memorize it” (Same attitude as most Physics professors—but is it true?)
Benefit of a “note-sheet” exam
Aids learning
•“can focus more on the concepts than just the equations and math” (many)
•“It's a very good study guide when studying for the exam.” (True, in my opinion! So why didn’t they do that to study for the closed note exam?)
•“As a professor once said to me, the physics is not in the formulas, but knowing how they are used” (Common Physics professor attitude)
More real-world
•“More practical in the real world”
Relieves stress (many)
•“You don't have to memorize simple equation which can allow for less stress.”
•“I personally was a lot more calm going into the [note-sheet] test.”
•“[provides] a security blanket for a little more confidence”
Better assessment of student learning
•“...a note card allows you to double check the equation you wrote down. Your brain can work in strange ways when it is under pressure and you can forget the most basic info and the note card just gives you that safety blanket”
Higher scores
•“You can get a higher score while knowing less physics”
•“Less deducted points for simple memory lapses”
No benefit to note-sheet exams
•“There are none [if the professor gives the hardest formulas on the exam]”
4.What are the implications of your work for your teaching and student learning and more broadly for the practice of teaching in your field?
Exam style is obviously a complicated topic, as evidenced e.g. by the students’ open-ended comments: some students felt closed-note exams more accurately assessed student learning; others felt the opposite. Some students felt closed-note exams more aided learning; others felt the opposite. So students’ perceptions of exam styles—and professors’ as well, undoubtedly—demonstrate a wide range. The complicated nature of “perception vs. reality” was also demonstrated in the students’ overall unhappiness about how well the exams—both exam styles!—accurately assessed their ability.
A few items were clear from this project: students did study more for the closed-note exam, and students taking the exam closed-notes did not perform any worse than students allowed the note-sheet. Thus any disadvantage in the taking of the exam without a note-sheet was offset by additional learning gained by studying longer for the exam.
Some items were not as clear: in preparation for the closed-note exam, students may have invested more time doing homework, and they may have paid more attention during class. But these effects were not as large, and were potentially ambiguous. Also, many students felt that in general more information would be retained in a closed-note exam, but when probed for details in how much information they themselves would retain, there was no difference between the closed-note and the note-sheet exams.
Overall, I would say that this project has yielded many interesting results which can serve as a springboard for further research in this area. There have been many prior papers published on the concept of open-note exams, but overwhelmingly they have been full of opinion and/or anecdotal evidence, and empty of concrete data. Hopefully the data I was able to obtain will thus add to the overall body of research in this area.
5.Describe how you have presented and/or will present the project to others (e.g., presentations on campus or at professional meetings, publications, open classroom, etc).
So far, the only formal presentation has been my poster at the OPID Spring 2007 Conference. Informally, however, I have had some discussions of the results with colleagues (see question 8, below, for one example). I am considering submitting these results for publication in an education research journal, but have not yet taken serious steps along that path.
6.Describe any products such as papers, publications, course portfolio, web-based materials, etc. you have produced related to the project.
None, yet.
7.Describe the next phase of your work on the project. If the project is not yet completed, what remains to be done? If it is completed, describe any plans to continue working on questions, problems, or issues related to teaching and learning.
This particular project is completed, but as noted above some things warrant further study—specifically the time spent on homework, and the attention paid during class. Those could be topics for a follow-up study. Also, the retention question is interesting—what the actual retention is like, rather than just the students’ guess of their retention.