ICT Prototyping Phase 8 Evaluation
REGISTRATION MARKS
12th September 2000
Executive Summary
The tests for this device did not yield a result that leaves the Museum happy with the current status of this programme. Although there was positive reaction from some visitors, especially people familiar with the concept of registration marks, it was felt that the programme needs to be targeted further at the specialist audience. Concurrently with the change in target, the dating process and the results given for each object dated need to further improve the experience of visitors who choose to use this programme.
Consequently, the concept of the device, its structure, labelling and process will be rethought in a paper prototype that will then mocked into a basic screen version. If this revised device is felt to be satisfactory, Oyster will be requested to build it for retesting in an environment that will mimic its final gallery setting as closely as possible.
Introduction
Brief for the Device
The Registration Marks terminal is a single application that will appear in the galleries. It will be primarily accessed in the Victorian Understanding Objects display.. The following objectives for the device are those taken from the original Evaluation Strategy for the ICT.
• To introduce the history and purpose registration marks and numbers.
• To allow visitors to look up a registration mark or number and find out the date of the design.
• To show in addition, using the objects in the Protecting New Designs display, the types of material about an object that were lodged at the Public Record Office when the design was registered, thus introducing visitors to the Design Registration system, how it worked then and how it can work for personal research nowadays.
• Through the web, to allow people to look up the date of the design of registered objets from home.
• To provide access to the PRO Web site through the terminal.
Brief and Overall Methodology for Evaluation
An initial round of public testing was conducted with the public between the 13th March and 16th June 2000. The analysis and evaluation of this testing yielded a number of changes to the design and content of the device and these were implemented by Oyster prior to a second round of testing. The Museum wished to assess whether the changes implemented had succeeded in tackling the usability issues that had arisen during the first round of testing. In particular:
• Test revised structure for visitor understanding.
• Check for usability of all buttons.
• Check usability of dating process.
• Check understanding of displayed objects page (PRO page).
Analysis of the evaluation focused on usability, content (audience relevance) and technical performance.
The second round of testing was conducted with samples of 10-15 people depending on the range of questions involved. The methodology was the same as for the first round of testing. Staff were not included as a separate sample of users for these tests.
Description of Prototype Device
The prototype device was constructed according to the ‘Brief for Phase 6 - Rebuild' section of the ICT Evaluation Report for Style Guide dated 20th June 2000.
The inclusion of mottoes into the bookplates after the typing a name activity was not included as this was felt to be only possible if the Museum felt that suitable raw bookplate content would allow it. It was also stated to be a highly complex technical problem for Oyster if mottoes need to be included in anyway other than in as horizontal text on the bookplate.
Description of Public and Staff Testing
Hardware and Software
Testing was conducted using the same prototype rig as for the first round of public testing: Two 15" screens were connected to PIII 500's under Windows NT with 128MB RAM. Screen resolution was 1024x768 with 24bit colour. Operation was via touchscreen.
Server architecture utilised a Sun Ultra5 running applications through Apache under Solaris 7. Database architecture was Oracle 8.0.6 with Java served through NewAtalanta ServletExec 2 and Sun JDK 1.2.1.04
Test conditions match those specified in the original System Architecture Definition Report. The exceptions to the definition were:
• Recent updates and patches to previously specified version of server architecture (JDK, ServletExec, Solaris, Oracle)
• Replacement of Netscape Enterprise Server to Apache
• Switch to a UNIX based server to increase the stability of Oracle database.
The software program was run through Internet Explorer 5.5 running in kiosk mode.
Methodology and Environment
The test was carried out in the Best of British Gallery. Each test screen was placed on a work surface with stools for visitors to sit on. Three objects with registration marks were placed in an adjacent case, with the marks reproduced on the object labels.
Testing was conducted by the Device Manager, Audience Research Manager and Multimedia Manager on 4th August.
A combination of observation and interview was used in order to see how visitors used the device without direction.
Description of Sample
VisitorsNumber of people / 15
Gender / Male = 8
Female = 7
Age / Under 25 yrs = 2
25-34 yrs = 3
35-44 yrs = 1
45-54 yrs = 6
55 plus yrs = 3
English as first language / Yes = 13
No = 2
Visitors were from UK (9), USA (3), Sweden (2), South Africa (1).
All visitors used a computer at home or work/college.
Two visitors were interviewed in a group of two.
Findings from the Public Testing
Usability
Problems with the touchscreen
Seven of 15 visitors appeared to have problems with the basic function of touchscreen technology. They needed prompting to touch the screen initially, and sometimes this continued throughout the programme. As this problem has not occurred to anything like this degree with any of the other touchscreen programmes it is suggested that in fact it may be more due to lack of clarity about what to do than with the touchscreen per se.
Dating an object with a diamond mark.
• Thirteen out of 15 visitors successfully dated an object with a diamond mark using one of the examples in the case next to the screen. Of these eight needed some help from the interviewer. Three of these needed constant help throughout.
• Amongst the help needed was how to use the diamond mark and how/why to look in the case for diamond numbers on labels/objects. Some visitors did not proceed pass the home page without prompting.
• Three visitors had difficulties reading the diamond off the object or label. One commented that the image on screen was not the same as that on the label.
• Two visitors needed prompting to press “Date a Design” once they had finished inputting the numbers and letters.
• The specialist and the under 25 year-old who were tested made no requests for help.
• One visitor used single digits to enter numbers of the day (e.g. 1 then 8 for 18 instead of pressing the number 18 button).
Dating an object with a registration number.
• Eleven out of 12 visitors successfully dated an object with a registration number using one of the examples in the case next to the screen. Three visitors did not attempt to date a registration number.
• Two visitors asked how to progress after entering the number and were prompted to press “Date a Design”.
• One visitor commented that the tester would not always be there to help and added “The programme was not one of the most exciting experiences of my life”.
Object page
• When asked what the object screen was all about only two out of 15 visitors stated that it showed the date of the object’s design, the picture of the Museum object, the picture of the design registration sample and a picture of the patent office register. Two visitors said that it showed the first three of these (i.e. they mentioned everything except the picture of the patent office register), and one mentioned the last three (i.e. everything except the date of the object design). One visitor mentioned the date of the object design only and one the picture of the patent office register only. This gave a total of just under half (seven of 15) who mentioned at least one of the things that the object screen showed. Two gave other comments that indicated some level of understanding: “It seems clear, the written record is hard to decipher”, and “Very useful as more close up. Written record is particularly interesting”. Two others just commented that it was clear or excellent without explaining and four others did not respond to this question. With hindsight the question asked to visitors was not always well understood and was sometimes modified during interviewing to “What have you found out form looking at this screen?”. Nonetheless, the findings indicate a lack of clarity about what this screen shows.
• Three visitors needed prompting to drag the boxes around the object images.
General Understanding of what registration marks were used for
• Three of 15 visitors understood exactly what registration marks were for, i.e. protecting designs. Four people thought they were for dating objects, while another five gave miscellaneous answers that often indicated some understanding, for example to protect makers or to find out date and batch and when an object was made. Three visitors either did not or could not answer this question.
Suggestions for improvements
Nine out of 15 visitors suggested things that could be improved in the programme. This indicates that there are still major problems with both content and usability. A full list of suggestions is given at Appendix one. The most significant suggestion (by three visitors) was that the programme needed to be changed if it were intended for non-specialists. More guidance was needed and also clarification about why they would be interested in it. Four visitors suggested no improvements and made a positive comment, one visitor suggested no improvements with a negative comment and one visitor suggest no improvements without any comment.
Content (Audience Relevance)
• All comments on content are included in the usability section above.
Conclusions from public testing
• A lack of clarity persists about how to use the Registration Marks programme. Visitors needed a lot of help, particularly with the diamond mark part. This suggests that overall navigation is still in need of improvement.
• The “Date a Design” button posed problems for visitors on both the diamond mark and the registration number sections and should be reviewed.
• The object page is not clearly laid out. Visitors very often did not appear to realise everything that was on it. There were also specific problems with the small images that visitors had to touch to change the large image and to move around within it. Some visitors had to be prompted to use them and the written record proved difficult to read in this format.
• The programme does not currently communicate the purpose of registration marks particularly well. It may be that in its final gallery setting, surrounded by relevant displays and other information, this ceases to be a problem. However, this is an unknown quantity, and it should be considered whether any improvements can be made within the programme.
There is a sense, expressed by some visitors, that the programme is aimed at specialists and is not intended for ordinary visitors. This was confirmed by the very positive comments of the one visitor known to be a specialist with an interest in registration marks. For many non-specialist visitors there appeared to be a lack of interest and engagement with the programme, making it a less than satisfactory experience. A decision needs to be taken about whether it is acceptable for a gallery computer interactive to be aimed at a relatively small segment of the audience.
Conclusions for Phase 9 - Evaluation of Prototyping Phase
• It is felt that the current device aims to be too much to too many. Consequently, the device needs a rethink of concept, design and navigation. This will also require some additional work by Oyster.
• The Series and Device Managers will therefore work towards a paper version of the revised programme that will address all matters of text, design and process. This paper prototype will be tested with educators for usability and with specialists for process. The revised device will work on the following main principles:
• Make it clear from the outset that the programme is aimed at a certain audience and may not hold much interest for other visitors.
• Keep the dating process as quick and simple as possible.
• Break up the end screens so that the actual process of dating designs yields a series of interesting screens based around the dating process.
• The paper version once approved will be mocked up into an on-screen device based on the existing Oyster style guidelines that work across all gallery interactives.
• Once the Museum is satisfied that a revised device is at a satisfactory stage to be passed to Oyster, it will be rebuilt and retested. The environment for retesting will try to match that of the final British Galleries more closely than previous tests since this was felt to have a large impact on visitor reaction
APPENDIX 1 - Quotes from Visitors and Observation Notes
Usability
Dating an object with a diamond mark
• What do you want me to do? [Tell her to touch screen. I need to inform her how to enter each piece of data working round diamond.]. Explain to me what this is meant to do. [I explain.]