S207 AL Tablet PC Trial

The use of Tablet PCsfor marking physics assignments

Jonathan Underwood* and Stuart Freake,

Department of Physics and Astronomy, The Open University

[ * now at Department of Physics and Astronomy, UniversityCollege, London]

December 2007

Abstract

Marking and commenting on electronic tutor marked assignments (eTMAs) requires a simple and efficient method for tutors to annotate students’ scripts. We report here on a pilot project that provided associate lecturers (ALs) for a Level 2 physics course with Tablet PCs and appropriate software that enabled them to input handwritten annotation, including equations and diagrams, to files containing students' assignment solutions. Surveys of ALs’ views indicated that this method of marking eTMAs is effective and acceptable to students and to a large majority of ALs.

1Introduction

The majority of Open University courses are assessed by a combination of tutor marked assignments (TMAs), which are submitted by students at intervals during the presentation of a course, and an end-of course examination. The TMAs are marked by associate lecturers (ALs, also known as tutors), who are based at different locations around the country and are each responsible for a group of about 25 students. The traditional method of producing and marking TMAs is paper based: a student writes their answers to the assignment questions on paper (or word-processes them and prints them out), they use the postal system to mail the assignment to their AL, who marks and comments on the assignment by annotating the student’s script and completing a cover form (the PT3 form). The AL sends marked assignments and PT3 forms to the Open University in Milton Keynes, where the marks are recorded in a database, a sample of scripts are copied so that each AL’s marking can be monitored, and the scripts are then posted back to students.This system involves shuffling large quantities of paper around the country and there are delays, expenses and inconvenience associated with the postal system and with the transcription of marks and the copying of assignments for monitoring. Now that OU students and ALs are expected to have access to a computer and the internet, some of the delays, costs and inconveniencecan be reduced by handling TMAs electronically.

The OU has developed a system for handling electronic tutor marked assignments (eTMAs), and is encouraging course teams and students to use this system wherever possible. The eTMA system allows students to submit electronic files containing their assignments over the internet to a file handling system on a server in Milton Keynes. ALs download the eTMA files for their students from the server, mark and comment on the eTMAs offline, and then return the marked scripts over the internet to the server in Milton Keynes, where marks are automatically recorded in the appropriate database and a sample of scripts is made available for monitors to download for monitoring. Students can then download their marked assignment from the central server. This system has many advantages for students, ALs and the OU over the paper-based assignment system that relies on the post office for moving the assignments around the country, but it also has some disadvantages.

Gibbs and Simpson (2004) established a list of conditions under which assessment effectively supports learning. A key finding of this work was that it is essential that “feedback is timely in that it is received by students while it still matters to them and in time for them to pay attention to further learning or receive further assistance”. This issue was also highlighted in the ‘seven principles of good practice in undergraduate education’ (Chickering and Gamson, 1987, 1991). The eTMA system in principle allows ALs to give more rapid feedback to students following the cut-off date for the assignment than does the paper-based system.

However, the benefits of rapid feedback could be more than offset if the time required for students to produce an eTMA were significantly longer than to produce a paper TMA, so that their time is diverted from learning physics. The benefits might also be offset if a longer time were required for ALs to mark eTMAs to the same standard as paper TMAs, so that the quality and quantity of feedback provided to students were reduced. Assignments for physics and astronomy courses (and courses in some other disciplines too) generally involve large amounts of mathematical notation, equations and diagrams. Many OU students do not have the skills required to incorporate equations and diagrams into a word-processed file, and for some of those who do have these skills the additional time required compared with handwriting an assignment is prohibitive. However, these students might be able to scan a handwritten TMA, combine the resulting image files into a single document and submit this electronically. When marking eTMAs, ALs also need to be able to insert equations and diagrams, andfor them the time required to do this is generally the major issue. Keyboard and mouse based tools, such as equation editors and line drawing programs, are much more time consuming to use compared with annotating a paper script with a pen.

The pilot project described in this report was set up to investigate ways in which ALs could mark physics eTMAs easily, and could incorporate equations and diagrams just as they would if they were marking a paper assignment.There were four main requirements that we wanted to meet.

• Students should be able to prepare their TMA answers using the tools that they find easiest, and should not have to purchase any proprietary software. Scanned handwritten submissions should be acceptable.

• ALs should be provided with only one program that they had to learn to use to mark and annotate assignments efficiently, including annotation with equations and diagrams, without incurring an increase in workload. The marking software should therefore be usable whatever the file formats submitted by students.

• The annotated file returned by the AL had to be readable by the student. Formats that would be difficult for some students to open are not acceptable.

• No content or information – including the mathematical content – should be lost at any stage of the process.

For this project werecommendedusing the pdf file format for marking, since it is possible to use free software that installs a ‘virtual printer’ that can ‘print’ from any program that the student uses to create a pdf file.The pdf file can be annotated with the Tablet PC pen using PDF Annotator software in the same way that an AL would annotate a paper TMA. The annotated file can then be saved in pdf format and returned to the student to be read using Adobe Reader

A second eTMA marking project within the Centre for Open Learning of Maths, Science, Computing and Technology has investigated the use of Tablet PCs for marking eTMAs for a Technology course using Windows Journal rather than PDF Annotator (Fisher, 2007).

Another CETL project was set up in 2007 to investigate use of digital pens for producing eTMAs. About 30 S207 students were provided with digital pens, so that they could submit handwritten assignments via the eTMA system (Gorfinkiel 2007).

2The project

In March 2006 the Physics Innovations Centre for Excellence in Teaching and Learning (πCETL) at the Open University agreed to fund a pilot project to investigate the use of Tablet PCs by ALs for marking assignments. All of the ALs who were tutoring a Level 2 physics course, S207 The physical world, were invited to participate in the project. Since the project was set up after the start of the course, participation by both ALs and students was optional, and 19 of the 24 ALs agreed to take part. The participating ALs were provided with a Tablet PC (Toshiba Tecra M4), with appropriate software installed, to enable them to add handwritten annotation to a student’s eTMA file.

Students were informed about the pilot project at the end of March 2006, and told about some of thebenefits of using the eTMA system. They were told that not all ALs were participating, that student participation was optional, and that they should discuss with their AL whether eTMA submission was possible and what formats were acceptable for their eTMA files. It was left up to individual ALs to decide how much encouragement they gave to students to submit their assignments electronically.

The timing of the approval of the project and of despatch of the Tablet PCs meant that electronic submission was not possible for the first assignment (TMA01) in 2006, but most of the ALs were prepared to accept eTMAs for the second and subsequent assignments (TMA02 – TMA07). In 2007, all students could opt to submit electronically, and all ALs were expected to mark and return eTMAs electronically.

2.1Marking procedures with the Tablet PC

ALs had a number of options for marking and commenting on eTMAs using the Tablet PC.

2.1.1Ink annotation of a pdf file using PDF Annotator

We decided to recommend the PDF file format for marking, since it is consistent with all four requirements listed at the end of Section 1. The Tablet PCs were sent out with the PDF Creator application installed (Chinery and Heindörfer). This application creates a ‘virtual printer’ that can be used to convert into pdf formatany file that can be opened and printed from the Tablet PC. The AL simply opens a student's eTMA file – which requires that an appropriate application is installed on the Tablet PC – selects Print and chooses PDF Creator as the printing device. 'Printing' the file then produces a pdf version of the file, which is stored in a location specified by the AL, rather than printing out on paper. PDF Creator allows the user to combine a number of different files into a single pdf file.

The AL then opens the pdf file from the PDF Annotator application (Grahl Software Design), which was also preinstalled on the Tablet PC. With the computer in the tablet configuration, the pdf file can be annotated by writing on the screen with the tablet pen in exactly the same way that a paper assignment would be annotated with a conventional pen, but with various advantages:

- it is easy to change pen colour and line width – so it's possible to colour code different types of comments;

- it is easy to erase or correct comments;

- it is easy to move or resize comments;

- it is possible to save comments as macros that can then be inserted easily elsewhere in the same document or in other documents;

- it is easy to insert blank pages for additional comments, or to append files (such as specimen answers).

When the AL has finished marking, they ‘melt’ the annotation so that it becomes a permanent part of the document, they save the file in pdf format and follow the standard procedures for returning the eTMA to the OU server. The student downloads the marked assignment from the server and opens the file using Acrobat Reader.

One facility that was not available in PDF Annotator in 2006wasfor the input of text using the keyboard. When this application was running, the keyboard was disabled, so even if the computer were in the laptop configuration rather than the tablet configuration, it was only possible to annotate a document using the tablet pen.However, an upgrade to PDF Annotator (version 1.5.0.108, released May 2007) now allows simple text to be keyed into a text box, and ALs were able to make use of this facility when marking the later TMAs for the 2007 presentation of the course. Mixing keyboard entry and tablet pen entry is not particularly easy, since it involves switching back and forth between using the laptop configuration and using the tablet configuration. Switching between the two configurations of the screen takes about 15 seconds, which is not an unreasonable overhead, but the switchover is somewhat clumsy and it is not clear whether the hinge mechanism for the screen is designed for such frequent use. An alternative procedure used by some ALs is to connect a USB keyboard to the Tablet PC, so that both keyboard entry and tablet pen entry are possible in the tablet configuration.

2.1.2Ink annotation using Microsoft Word

A large majority of the S207 eTMAs are submitted as Word documents, and so a second option for eTMA marking is for ALs to use the Ink tools available in the Tablet PC version of Word 2003, which was installed on the computers that were provided.In contrast to the PDF Annotator approach described above, employing Word for marking requires that the student submits the TMA in a file format compatible with the version of Word used by the AL. Problems with equations can arise when a document produced in Word 97 is opened in Word 2003, though no ALs reported experiencing any such problems.

The Ink Annotation tool can be used to write with the tablet pen anywhere in an open Word document, and, as with PDF Annotator, it is possible to change pen colour and line width, to erase ink marks, and to reposition and resize comments. It is also possible to usethe Ink Comment toolto insert comments in the margin. This is similar to using the normal Insert Comment feature in Word, except that the comments are handwritten in a balloon in the right marginusing the tablet pen. These comments can include equations and diagrams.

When the AL has finished marking the script, the file is saved in .doc format for return to the student via the eTMA system.

Keyboard entry is always available while Word is running, so it is possible to switch back and forth between using the keyboard to enter simple text in the laptop configuration and using the tablet pen to enter equations and figures in the tablet configuration. Again, combining keyboard entry and tablet pen entry is much more straightforward if an external keyboard is used with the Tablet PC.

2.1.3Ink annotation using Windows Journal

A third way to mark eTMAs with a Tablet PC is to convert the student’s file to a Windows Journal file and then to annotate this file using the tablet pen. As with PDF Creator, Windows Journal installs a virtual printer with which any printable file submitted by a student can be converted to a Windows Journal file (.jnt), and this file can then be annotated in much the same way as a pdf file is annotated using PDF Annotator. This option was not suggested to S207 ALs, and none of them apparently discovered or used this method, but this was the approach used in a COLMSCT Tablet PC trial with a Technology course(Fisher, 2007).

3Evaluation

After completion of the 2006 presentation of S207, a brief questionnaire was emailed to the ALs who had participated in the pilot project, and 15 of the 19 participants responded. A more comprehensive questionnaire was emailed at the end of 2007 to the 24 ALs involved in that year’s presentation, and 23 of them responded. For both years, information about the number of paper TMAs and eTMAs marked by each AL for each of the course assignments was obtained from the assignment handling database.

Number of eTMAs submitted The number of eTMAs submitted by S207 students and their percentage of the total number of TMAs submitted have grown steadily, as shown in Figures 1 and 2. It is noteworthy that whilst the number of paper TMAs decreased by about 50% during each presentation, the number of eTMAs initially increased each year and only decreased slightly for the last few TMAs. However, we cannot say whether the relative stability of the number of eTMAs submitted in the second half of the course is due to better retention of students who use the eTMA system or to students moving from paper submission to electronic submission tocounteract drop out by eTMA users. At the end of 2006 25% of TMAs were being submitted electronically, and this percentage had increased to almost 40% by the end of 2007.

Figure 1 Number of eTMAs and paper TMAs submitted for each assignment during 2006 (shown on the left half of the chart) and 2007 (shown on the right).

Figure 2 Percentage of TMAs submitted electronically for each assignment during 2006 (shown on the left) and 2007 (shown on the right).

There was a wide variation in the use of eTMAs by different tutor groups. Figure 3 shows the numbers of ALs receiving different percentages of eTMAs. In 2006 12 of the 19 ALs received less than 20% of their TMAs electronically and only two received more than 40% as eTMAs. In contrast, the 2007 data show only five of the 24 ALs received less than 20% eTMAs and 12 received more than 40%.

Figure 3 Number of ALs receiving different percentages of eTMAs in 2006 and 2007.

One possible explanation for the disparities between the submissions of eTMAs by different tutor groups is that ALs varied in their stance towards eTMA submission. In the 2007 questionnaire, ALs were asked about this, and most of them said that they took a fairly neutral stance, indicating advantages and disadvantages of e-submission, but also indicating that they were happy to accept either form and it was up to individual students to decide. There is some evidence that AL who encouraged students to try the eTMA system, or who indicated a personal preference for electronic marking received a higher percentage of eTMAs: the AL who received 61% of TMAs electronically “said I’d prefer eTMAs, and emphasised that the choice was theirs”, whereas an AL who only received three eTMAs did not say anything about the different submission routes to their students. However, there were also ALs who encouraged e-submission but with little response, and those who were neutral but received a high percentage of eTMAs.