Logo above: University of Birmingham

Evaluation of alternative exam presentation methods for pupils with low vision

Report for RNIB

September 2010

Authors:

Dr Graeme Douglas

Dr Mike McLinden

Annette Weston

Visual Impairment Centre for Teaching and Research (VICTAR)

School of Education

University of Birmingham

Birmingham, B15 2TT


Contents

Acknowledgements 4

Executive summary 5

Summary of findings 5

Recommendations 7

1 Introduction and context 9

2 Student performance and views of alternative examination papers 10

2.1 Introduction 10

2.2 Method 10

2.3 Results 13

3 Case examples: Students who prefer very large hardcopy print (N24+) 23

3.1 Introduction 23

3.2 Method 23

3.3 Results 26

3.4 Discussion 37

4 In-depth interviews with staff involved in material modification 40

4.1 Introduction 40

4.2 Method 40

4.3 Analysis of results 43

5 Group Interviews: Students who are non-print readers 68

5.1 Introduction 68

5.2 Method 68

5.3 Results 71

5.4 Discussion 82

6 Technical analysis of the RNIB E-formatting software 83

6.1 Modifying the examination paper 83

6.2 Using the software to format the examination paper and quality of the output. 85

6.3 ‘Accessible’ files 87

7 Discussion 89

7.1 Summary of findings 89

7.2 Classroom practice versus public examination system 91

7.3 Possible solutions 92

7.4 Recommendations 94

7.5 Functional specification of future software 95

8 References 100

9 Appendix 1 Student Questionnaire 101

10 Appendix 3 Font size questionnaire 104

11 Appendix 3 Professional Interview Schedule 106

Acknowledgements

We would like to thank all our participants for generously giving us their time and views. We thank RNIB for funding this research. We thank the steering group for their helpful advice throughout this project. Thank you to Rory Cobb for general advice and help with the group interviews with students. Thank you to Sue Keil and Paul Nisbet for their general advice.

Executive summary

This is a report commissioned by RNIB in relation to examination access by blind and partially sighted learners. The research particularly focussed upon the evaluation of a prototype piece of software – ‘RNIB E-formatting software’. RNIB E-formatting software was designed to allow the teacher/school to modify examination papers in terms of font size and style, text and background colour, and the line width inside diagrams. Once it has been modified to meet the individual requirements of the candidate, a paper can be printed for them to have as a paper copy or it may be read on-screen.

The research team carried out five related pieces of research:

·  Student performance and views of alternative examination papers

·  Case examples: Students who prefer very large hardcopy print (N24+)

·  In-depth interviews with staff involved in material modification

·  Group Interviews: Students who are non-print readers

·  Technical analysis of the RNIB E-formatting software

Summary of findings

1. How do the Modified Large Print (MLP) versions of papers produced by RNIB E-formatting software compare to traditional MLP papers? (both process and outcome)

a.  Findings from a trial involving 21 students with low vision suggest that examinations that are formatted using the RNIB E-formatting software neither significantly advantages or disadvantages student performance compared with existing MLP approaches. Familiarity with examination format appears to be a more important variable.

b.  Professionals interviewed were broadly positive about the RNIB E-formatting software:

·  They were positive about having a greater range of options for examination formats which the software could offer.

·  They particularly identified control over page breaks and formatting of pictures/diagrams as key areas for further development.

·  There were mixed feelings about how the process of formatting and production could be incorporated into the short one hour period before the examination. Some had concerns about technical problems and working under time pressures.

c.  There is evidence that appropriate and consistent modification of examinations is of key concern to professionals irrespective of the method of delivery. The process of producing the software versions of the examination papers (from four different examination boards) appears to have an important associated benefit, namely standardising of the modification process and the general format of the examination papers.

d.  Case examples of five students who preferred very large print (greater than 24 point) demonstrated that they appear to benefit from hard copy large print formats which are different from those available through existing MLP approaches. An estimated 9% of GCSE-aged students with a visual impairment in England and Wales use font sizes greater than 24 point.

2. What methods of accessing examinations are useful for the future, and can the RNIB E-formatting software (or a developed version of it) play a useful part? (e.g. electronic copies, interactive papers)

  1. When reflecting on the ‘perfect scenario’, many professionals reported that students should have access to examinations which had been modified appropriately and could then be formatted in a flexible way to meet individual needs. The use of technology was seen as an important part of the formatting solution by all participants (teachers, students with low vision and non-print readers). This suggests a distinction between appropriate modification of a paper ‘at source’ and the formatting options that could potentially be provided by the software.
  1. Access to on-screen interactive examinations was seen by both students and professionals as offering a range of opportunities for improving access. Nevertheless, further work is required to enable these opportunities to be developed further. Key challenges identified through discussion with students (blind and low vision) as well as professionals included:

·  technical work is required to ensure computer-based material is fully accessible;

·  appropriate examination modification is carried out ‘at source’;

·  training of students in the use of assistive technology to access exams;

·  training of staff to teach students how to use this assistive technology for exams access.

  1. Future development needs to be flexible enough to offer a range of on-screen formats (e.g. a range of fonts, sizes, colours, and zoom facilities). They also need to include options for hard copy versions of questions and/or supporting information such as diagrams and tables to be produced in a variety of formats.

Recommendations

We present two broad recommendations based upon the research findings. The recommendations are made in relation to the project research questions, i.e. particularly linked to the potential of the RNIB E-Formatting software (or similar) in the provision of accessible examination papers. It is important to note that the authors advise that the recommendations are implemented as part of a more general review of examination and curriculum access approaches for visually impaired students in England and Wales, specifically in relation to:

·  The procedures and quality control of examination provision (including the process of modification);

·  Curriculum access strategies adopted in the classroom (particularly in relation to the use of low vision aids and computers).

Recommendation 1:

A software application for supporting the production of an increased range of examination formats for students with visual impairment should be developed. The application would be used by teachers and visually impaired students, and it should have the following functions:

a)  a range of formatting options;

b)  printing facilities for students who require hardcopy large print examinations;

c)  onscreen presentation facilities for students who require onscreen access to examinations.

The report provides a draft functional specification for the software application. The RNIB E-formatting software evaluated in this research project provides a very useful prototype of such an application.

Recommendation 2:

The future development of mainstream interactive computer-based examinations presents significant opportunities for improving access to examinations amongst visually impaired students. To ensure that visually impaired students can benefit from these opportunities the following general recommendations are made:

a)  Examination boards and designers of computer-based examinations technology should work with access technology experts (for example, within RNIB) to ensure that inclusive design principles are incorporated into the development of this technology.

b)  Developments in computer-based examinations will mean that educators must continue to ensure that visually impaired pupils are taught appropriate access skills so that they can independently access these examinations.

1. Introduction and context

This is a report commissioned by RNIB in relation to examination access by blind and partially sighted learners. The research was designed in 2008 in response to the Research Brief prepared by RNIB Corporate Research Team: “Passing the test: how accessible are public examination systems for blind and partially sighted pupils in the UK”. The research was carried out in two phases of work carried out between April 2009 and July 2010:

·  Phase 1: International comparison of approaches

·  Phase 2: Trialling and functional specification of the RNIB ‘E-formatting’ software

Phase 1 report (entitled ‘Summary report on international systems of exam access for visually impaired pupils’) was presented in September 2009. This report presents the findings from Phase 2. Following some reconfiguration of the project following the Phase 1 report, Phase 2 had the following research questions:

1.  How do the modified large print (MLP) versions of papers produced by RNIB software compare to traditional MLP papers? (both process and outcome)

2.  What methods of accessing examinations are useful for the future, and can the RNIB software (or a developed version of it) play a useful part? (e.g. electronic copies, interactive papers)

The research team carried out five related pieces of research which are presented in turn:

·  Student performance and views of alternative examination papers (linked to research questions 1 and 2)

·  Case examples: Students who prefer very large hardcopy print (N24+) (linked to research questions 1 and 2)

·  In-depth interviews with staff involved in material modification (linked to research questions 1 and 2)

·  Group Interviews: Students who are non-print readers (linked to research question 2)

·  Technical analysis of the RNIB E-formatting software (linked to research question 1)

2. Student performance and views of alternative examination papers

2.1 Introduction

The research team carried out an experiment which compared paper-based formats prepared using existing large print approaches (modified within the existing large print ‘access arrangement’ policies) to formats prepared using the E-formatting software. The aim was to examine whether either format gave any measurable advantage to the visually impaired pupils (in terms of time taken). The participants also gave their opinions about the different formats through a short questionnaire and group interviews.

2.2 Method

Overall design

A repeated measures design was employed so that each participant tried out two alternative presentations of an examination – modified large print (‘MLP’) and E-formatting software (‘Software’). Different versions of the examination papers and order of presentation were balanced. Time taken to complete the tasks was measured, and opinions about the different formats were gathered through a questionnaire.

Participants

Twenty one participants took part in the study in three separate sessions. All were taking part in a study week at Worcester New College during their Easter holiday in 2010. All had low vision, were print readers and expected to be using print-based examination papers in the GCSE examinations. The majority of participants attended mainstream schools (except one mainstream school with resource base and one ‘private’). A breakdown of the participants is presented in Table 1.

Table 1. Characteristics of the participants taking part in the trial (N=21). Note: frequencies based upon available data.

Characteristic / N
Age group
Year 10 / 9
Year 11 / 12
Gender
Male / 13
Female / 8
Font size (provided prior to testing)
12 point / 2
18 point / 10
24 point / 9
Font size (questionnaire response)
16 / 1
18 / 8
24 / 8
36 / 1
LVA used (questionnaire response)
Yes / 8
No / 12
Preferred font (questionnaire response)
Arial / 9
Impact Bold / 1
Comic Sans / 3
Any / 1
TOTAL / 21

Materials and variables

Content for two ‘pretend’ examinations was created (version A and B). Both examinations had the same structure and length and were matched for content difficulty. Each paper consisted of ten questions. The question styles were drawn from existing papers and included:

·  Multiple choice with tick box (‘Tick the BEST answer’)

·  Sorting statements into categories

·  Multiple choice (Put a ring around the correct answer)

·  Reference to a picture, single word written answer on answer line

·  Reference to a table, multiple choice

·  Labelling a diagram

·  Reference to a graph, single word written answer on answer line

·  Reference to text and a picture, single word or two word answers

·  Linking of associated statements

As part of the introduction given in each session, it was emphasised to participants that we wanted them to complete all questions and that we were testing the quality of the material and not testing them. For this reason the questions were very easy (containing relatively simple general knowledge and logical questions). The key measured variable was time taken to complete each examination paper.

After completing the two versions of the examinations (as described in the procedure below), participants completed a questionnaire (see Appendix). The questionnaire contained questions in relation to:

·  Personal working preferences (font style and size, use of LVA)

·  Preferences for different formats of examination and explanations

·  Ideal examination format and potential of onscreen examinations.

On completion of the questionnaire participants took part in semi-structured group discussions. The topics explored were:

·  Reflections upon the versions of the examinations just taken

·  General experience and expectations in relation to taking examinations

·  Potential of computer-use in examinations.

Procedure

Participants were allocated into three groups and each data collection session took place over approximately a 1.25 hour period. At the beginning of the session the broad purposes of the research was explained and it was emphasised that the material was being tested rather than the participants themselves. Participants were then asked to sign a consent form declaring that they were willing to take part in the research, but it was noted that they could withdraw at any time. Participants were each given a £10 gift voucher irrespective of whether they chose to take part in the research or not. It is worth noting participants were taking part in a study weekend in preparation for GCSE examinations and taking part in this session was seen as a useful and relaxed way of practicing and considering different formats of examination papers.