The impact of on-line revision on GCSE results
by Karen Osborne
Paper presented at the British Educational Research Association Annual Conference, Institute of Education, University of London, 5-8 September 2007
Table of Contents
Section / PageAbstract / 2
Background and Data Sources / 3
Definitions / 5
Population Summary / 6
Summary of Usage / 8
Outcomes / 10
Contextual / 11
Hours of Use / 13
Prior Attainment / 14
Conclusions / 21
Further Work / 22
Appendix / 23
Abstract
This paper reports the findings of a quantitative investigation into the impact of an on-line revision tool, SAM Learning, on GCSE results in several Local Authorities (LAs). SAM Learning is an online subscription revision service for GCSE and SATs in English secondary schools and is used by more than half of English state secondary schools.
Students access e-learning via lessons and revision sessions in schools as well as at Saturday revision sessions organised by the Local Authority. The revision sessions are targeted towards pupils residing in Neighbourhood Renewal areas and those more likely to need additional support. This research was commissioned by the LAs from the author (an independent consultant) in order to determine the effectiveness of the revision tool.
- Is using the tool associated with higher GCSE attainment?
- Are students using the tool more likely to be already motivated, have better access to the service during out of school hours, have higher prior attainment and/or come from a more advantaged background than those not using SAM Learning?
- After accounting for the above differences, do pupils using the tool make more progress between KS3 and GCSE?
The research sought to determine the effect of SAM Learning on the results of Year 11 GCSE students in 2005/06. Data was collected on more than 11,500 students attending around 50 schools in four LAs. SAM Learning provided data on the extent to which the students have used the tool, and this, at pupil level, was combined with the GCSE results and the Key Stage 3 and Key Stage 2 results of the pupils as provided by the LAs. Regression analysis was used to control for socio-economic factors and prior attainment in order to explore the effects of the tool on pupil progress between KS3 and GCSE.
Summaries of the key GCSE indicators showed that pupils who used e-learning achieved more, or higher graded, GCSEs than pupils who did not. Pupils spending up to 2 hours using the service achieved, on average, one more GCSE graded A*-C than those who did not use the service at all.
This was investigated further by looking at the GCSE results by socio-economic factors, and where information on prior attainment was available, this was used to determine how much progress had been made by different groups of students specifically comparing those who had used e-learning with those who had not.
The findings show that the difference in performance between those who used the service and those who did not, is greater between boys than girls indicating that e-learning appeals more to boys than it does girls. Pupils who are eligible for Free School Meals are more likely to achieve 5 or more A*-C grades if they use e-learning as are pupils who have English as an additional language.
Pupils achieving an average level of less than 5 at Key Stage 3 were almost twice as likely to achieve 5 or more A*-C grades at GCSE if they used e-learning than those who did not use the service. Additionally, the value added from average Key Stage 3 level to average GCSE total points was uniformly high for pupils using e-learning regardless of their Key Stage 3 starting point.
Because other (unmeasured) factors may differentiate those pupils using the system from those who did not, a full Randomised Control Trial would be necessary to conclude that the online revision tool improves performance. However the results are positive and promising.
Background and Data Sources
Research has shown that the use of ICT in teaching and learning is a positive motivator and produces improved outcomes at GCSE and Key Stage 3 in certain subjects, specifically for lower and middle ability groups. There are no research results found by this author that show use of ICT and e-learning as having a negative impact.
…motivation has been enhanced most positively in school situations where ICT is used within a framework that considers impacts upon learning, teaching, and the management of learning and teaching (Passey, D and Rogers, C; 2004)
For each core subject at key stage 3 and at GCSE level as a whole, there is a positive relationship between better ICT learning opportunities and higher pupil achievement (Becta 2003)
…ICT has positive effects on attainment outcomes(ImpacT2; 2002)
…well constructed use of ICT can enhance students' GCSE performance (Rivett, R; 1999)
The use of ICT, through independent learning facilities, allowed students greater control over their learning and encouraged the development of new ways of learning, both autonomously and co-operatively (Cunningham, M., Harris, S., Kerr, K. & McEune, R. 2003)
The research seeks to determine the effect of using an on-line revisiontool on the results of Year 11 GCSE students in 2005/06.
SAM Learning is an online subscription revision service for GCSE and SATs in English primary and secondary schools. SAM Learning Secondary is the service chosen by more than 800,000 students to support their revision. Over 60% of all English state secondary schools are current subscribers with more than half of all Local Authorities as key partners.
The e-learning tool offers over 60 topics for SATs, GCSE and A Level.Students study at school and at home, so are able to access e-learning via lessons and revision sessions in schools as well as at home and at Saturday revision sessions organised by the Local Authority.
Revision sessions are, in one LA, specifically targeted towards pupils residing in Neighbourhood Renewal areas and those more likely to need additional support.The funding for the first subscription to SAM Learning in this LA was provided as part of the Neighbourhood Renewal project and after three years, schools were asked to fund the next subscription.
The independent research was commissioned bythis LondonLA in order to determine the effectiveness of e-learningand therefore the validity of funding the subscription. Other Local Authorities also commissioned the research and this report concentrates on the results of four;three of which are in London and the fourth is in the Midlands.
This paper reports the findings of a quantitative investigation into the impact of the use of e-learningfor revision on GCSE results in these fourLocal Authorities.Data was collected on more than 11,500 students attending 58secondary schools in four LAs.
The results of this study are entirely independent of SAM Learning Ltd.
Definitions
A pupil is defined in this report as someone on roll at a secondary school in one of the participating LAs in January 2006. They will have been being taught in Year 11 though they may or may not have been studying at the end of Key Stage 4. The contextual information would have been collected for each pupil from schools by the LA in January 2006 as part of the School Census (PLASC) collection.
GCSE results are as provided by the LAs at pupil and subject level. Pupils who were on roll in but had recently arrived from overseas, did not speak English as a first language and had not previously attended a school in England, are included in the results where they were provided by the LA. GCSE results were matched (where this was not already done by the LA) to their contextual information.
A user of e-learning is defined as a pupil, as described above, who logged on and engaged with a task for more than zero hours. Time spent logged in but not working is not counted. Once logged on, a user can revise any subject at any key stage regardless of their own curriculum level. It is assumed that even if a user engages with work at a lower level than their curriculum level, that this will still be a benefit in terms of revision. Time spent learning is described as ‘task hours’.
SAM Learning provided data on the extent to which pupils in each LA used the tool. In order to gain an understanding of the types of pupils who use e-learning, this was matched to the contextual information for each pupil level as collected from schools.
This report does not investigate the subjects or levels of work studied; it concentrates on the length of time spent using e-learning by individual users.
Population Summary
In 2006, 52.6% of Year 11 pupils in theLAs spent some time using e-learning. The proportion of pupils in individual schools spending time using the tool ranged from 0.0% to 93.5%. Across the LAs, a greater proportion of girls (55.3%) than boys (50.0%) spent time using e-learning indicating that e-learningis more accessible for girls. This varies across the LAs as seen below; where one authority shows more boys using e-learning than girls:
Table A: Number and percentage of pupils using SAM Learning in Year 11 in 2006, by gender and LA
Total Pupils / All e-learning usersAll / Boys / Girls / All / % / Boys / % / Girls / %
LA 1 / 3,663 / 1,840 / 1,823 / 2,210 / 60.3 / 1,021 / 55.5 / 1,189 / 65.2
LA 2 / 2,560 / 1,276 / 1,284 / 1,327 / 51.8 / 633 / 49.6 / 694 / 54.0
LA 3 / 3,774 / 1,954 / 1,820 / 2,034 / 53.9 / 1,017 / 52.0 / 1,017 / 55.9
LA 4 / 1,692 / 952 / 740 / 576 / 34.0 / 340 / 35.7 / 236 / 31.9
All LAs / 11,689 / 6,022 / 5,667 / 6,147 / 52.6 / 3,011 / 50.0 / 3,136 / 55.3
In one Authority, as many as 60.3% of pupils used e-learning compared with 34.0% in another LA. In total, over 6,000 Year 11 pupils were accessing on-line revision in 2006 in these LAs.
In each of the LAs, the proportion of pupils eligible for Free School Meals (FSM) was lower among those using e-learning than in the whole population, indicating that pupils deemed as more affluent are more likely to use the tool to revise; that an on-line revision tool is less accessible for FSM pupils, or that the FSM pupils are less motivated to use such a tool.
This is probably because most revision is done outside of school hours using a PC at home and a pupil entitled to Free School Meals is less likely to have access to a home PC.
Table B: The percentage of pupils eligible for FreeSchool Meals in Year 11 in 2006, by LA
% Pupils eligible for FreeSchool MealsAll Pupils / Pupils using e-learning / % point Difference
LA 1 / 9.8 / 7.6 / 2.2
LA 2 / 27.2 / 24.6 / 2.6
LA 3 / 16.2 / 13.8 / 2.4
LA 4 / 23.9 / 22.9 / 1.0
All LAs / 17.7 / 14.8 / 2.9
The table above shows that the percentage of FSM pupils using e-learning is 2.9 percentage points lower than the percentage of FSM pupils in the whole population.
The opposite appears to be true for pupils whose first language is not English. A greater proportion of pupils with English as a second language usee-learning than are in the population.
Table C: The percentage of pupils with English as a second language in Year 11 in 2006
% Pupils with English as a second languageAll Pupils / Pupils using e-learning
All LAs / 15.9 / 16.3
Pupils from different ethnic backgrounds usede-learning in different proportions.Pupils from a white British or an Asian background were more likely to use the tool whereaspupils from black or other backgrounds were less likely to use it.
Table D: The percentage of pupils from different ethnic backgrounds in Year 11 in 2006
% Pupils from different ethnic backgroundsAll Pupils / Pupils using e-learning
White British/Irish / 65.1% / 68.1%
Other White / 3.3% / 2.9%
Black/Mixed Black / 14.4% / 12.4%
Asian/Mixed Asian / 9.5% / 10.3%
Other/Unknown / 7.4% / 6.2%
This differs, again, across the four LAs. One authority saw a greater proportion of pupils from black African backgrounds using e-learning than you would expect from the population. Some of the ethnic groups are too small to draw significant conclusions from.
These distributions were the same for other contextual factors such as pupils with Special Educational Needs and Children in Care. Pupils deemed as deprived or in a minority based on their contextual characteristics were less likely to use the revision tool. The Appendix provides a full breakdown.
Summary of Usage
The number of task hours of use is recorded. Pupils in the LAs in this report spent, on average, 8.5 hours using e-learning. Across the four Local Authorities, boys spent less time on average (8.3 hours) using the tool than the average time spent by girls (8.6 hours) though this varied considerably across the LAs:
Table E: The average length of time spent by pupils using e-learning in Year 11 in 2006, by LA
Average task hours of e-learningAll / Boys / Girls
LA 1 / 9.2 / 8.5 / 9.7
LA 2 / 8.7 / 9.1 / 8.3
LA 3 / 8.2 / 8.2 / 8.2
LA 4 / 6.0 / 6.3 / 5.5
All LAs / 8.5 / 8.3 / 8.6
Pupils in one school had an average of 15.3task hours, the highest of all the schools in the four Local Authorities.
The expectation, for Year 11 pupils using SAM Learning, is that there will be an increase of at least one grade if they revise for more than 10 hours. In the four LAs more than a quarter of pupils (27.3%)had more than 10 task hours and girls were more likely than boys to put in these hours. 13 of the pupils in the LAs used the tool for more than 100 task hours each.
Table F: Number and percentage of pupils using e-learning for more than 10 hours in Year 11 in 2006, by gender
Users with more than 10 task hoursAll / % / Boys / % / Girls / %
1,678 / 27.3 / 803 / 26.7 / 875 / 27.9
As well as recording the number of hours spent on-line, the number of hours spent revising outside of school hours (after 3:30 and at weekends) was also recorded. This allows analyses of those who were able to access the tool at times other than at school, including Saturday revision centres and those revising at home.
Table G: The number and percentage of pupils using e-learning outside of school hours in Year 11 in 2006, by LA
Users outside of school hoursAll / % / Boys / % / Girls / % / FSM / %
LA 1 / 1,650 / 74.7 / 780 / 76.4 / 870 / 73.2 / 103 / 61.7
LA 2 / 936 / 70.5 / 427 / 67.5 / 509 / 73.3 / 213 / 65.1
LA 3 / 1,279 / 62.9 / 637 / 62.6 / 642 / 63.1 / 137 / 48.8
LA 4 / 371 / 64.4 / 227 / 66.8 / 144 / 61.0 / 84 / 63.6
All LAs / 4,236 / 68.9 / 2,071 / 68.8 / 2,165 / 69.0 / 537 / 59.2
In each authority, between two-thirds and three-quarters of pupils who used e-learning did so outside of school hours. On average, 68.9% of users were revising on-line outside of school hours at some point.
Pupils eligible for Free School Meals were less likely to use the revision tool outside of school hours with 59.2% doing so, implying that there is perhaps less access to the service for these pupils.
Of those pupils who used e-learning and did so outside of school hours, they spent on average, 7.2 hours using the service during those hours.Pupils eligible for Free School Meals had only 5.5 task hours outside of school hours. This may be during the Saturday revision sessions since they were aimed at pupils in more deprived areas.
Table H: The average hours spent using e-learning outside of school hours by users in Year 11 in 2006, by LA
Average hours spent revising on-line outside of school hoursAll / Boys / Girls / FSM / Not FSM
LA 1 / 8.1 / 7.4 / 8.8 / 5.5 / 8.3
LA 2 / 7.0 / 7.6 / 6.5 / 6.0 / 7.3
LA 3 / 6.2 / 6.0 / 6.4 / 4.6 / 6.4
LA 4 / 6.5 / 6.7 / 6.1 / 5.9 / 6.7
All LAs / 7.2 / 6.9 / 7.4 / 5.5 / 7.4
Outcomes
In order to determine the effectiveness of this type of service, the following questions were proposed:
- Is using the tool associated with higher GCSE attainment?
- Are students using the tool more likely to be already motivated, have better access to the service during out of school hours, have higher prior attainment and/or come from a more advantaged background than those not using e-learning?
- After accounting for the above differences, do pupils using the tool make more progress between KS3 and GCSE?
The contextual information provides a description of the type of pupil who uses the on-line revision service. The information provided by SAM Learning and by the LAs was combined,at pupil level, with the GCSE results and the Key Stage 2 and 3 results of the pupils as provided by the LAs.
Summaries of the key GCSE indicators show that pupils who used e-learning achieved more, or higher graded, GCSEs than pupils who did not. The chart below demonstrates the difference between the proportion of pupils who achieved 5 or more A*-C grades at GCSE comparing those who used e-learning and those who didn’t.
Chart 1: The percentage of pupils achieving 5 or more A*-C grades at GCSE in Year 11 in 2006
At a simple level, it appears that those who used e-learning were more likely to achieve the expected level at GCSE.
Table I: The percentage of pupils achieving 5 or more A*-C grades at GCSE in Year 11 in 2006, by use of e-learning
All e-learning users / Not e-learning users / All pupils%5+ A*-C / % 5+ A*-C / % 5+ A*-C
66.2% / 43.5% / 55.4%
This was investigated further by comparing the GCSE results by socio-economic factors to determine the results of different groups of students in relation to their use of SAM Learning.
Contextual
The chart below shows that the difference in performance between those who used the service and those who did not, was greater between boys than girls indicating perhaps that while e-learning appeals more to girls than boys in terms of uptake, boys benefit more in terms of outcomes than girls.
Chart 2: Percentage of pupils achieving 5 or more A*-C grades at GCSE in Year 11 in 2006, by gender