Changes in the Derived Grades Procedures Starting in 2003

Changes in the derived grades procedures starting in 2003

1Introduction

In 2003 changes in the Derived Grades procedure will become operational. The purpose of this procedure is twofold. One purpose is to adjust the external assessment result for candidates whose performance in the external assessment is lower than estimated and for whom there would be a strong probability of a successful appeal. The other purpose is to assist in determining an external result for absentees.

The computerised Derived Grades procedure, introduced in 1991, replaced the first phase of the former assessment appeals and of absentee and adverse circumstances procedures. This involved examiners in scrutinising the Order of Merit (the list of all candidates and their external results, sorted by estimate) and deciding whether the centre was a good estimator of performance. This lengthened the time needed for the appeals results, even for clear-cut cases. Another issue was the effort required ensuring that this scrutiny took place in the same way across all subjects. The automatic procedure was created to

provide a solution before certification
save examiners’ time
treat candidates in all subjects in the same way

The procedure has been expanded and improved several times after the method was computerised in 1991 for Standard Grade. By 1995, it was extended to the SCE Highers, but with two major differences. Firstly, bands were used instead of grades, and secondly, higher criteria for the agreement between estimates and external results were set than for Standard Grade, because of the increased stakes and greater externality at the higher level.

In 2001, it became evident that the old procedure for SCE Higher should not be applied to the new National Course awards. It was based on criteria which favoured small entry groups, which used to be exceptional. The new National Courses, however, with the wider range of subjects and levels, comprises many courses with just a few candidates per centre. It was therefore decided to replace the original criteria with one fixed criterion for all. At the same time the existing correction for severe centres was abandoned. For Standard Grade the procedure remained unchanged. The new procedure and criterion for National Courses were applied to the results in 2001 and 2002. It was made clear that the procedure would be monitored and reviewed in 2002.

For 2003, Standard Grade entries and results will be processed ‘on the same platform’ as all the other qualifications. This means that the old programs have had to be rewritten. This seemed to be the right moment to review the Derived Grade procedures for both Standard Grade and National Courses. They needed to be simplified and made more transparent, and apply the same principles, while respecting differences between the two qualifications, such as the use of elements and bands, and between their appeals processes.

The following paragraphs describe the underlying principles and the new rules. They also mention the main changes and what their implications might be. A prototype of the new format of the external report is attached.

2Principles underpinning the new procedure

The procedure consists of two basic rules, one for Standard Grade, and one for National Courses. These rules are based on the following principles.

Simplicity

The procedure needs to be simple to explain and apply. It should be easy for centre staff to understand the rules and even apply them. The new report will be easier to understand as well and will provide the outcomes of the procedure (instead of the initial external results before grades were derived). It will also list the candidates who have been awarded improved grades.

Dependable estimates

The procedure provides improved external results and helps determine a result for an authorised absentee depending on the centre’s estimates. As this happens in an automatic process, it needs to be clear without any doubt that the estimates are dependable. To be dependable they need to meet two criteria. Firstly the centre’s estimates for other candidates need to agree sufficiently with their external results. Secondly, there need to be enough estimates to be sure that the agreement is not due to chance.

Relevant and minor changes

As the main purpose of the procedures is to anticipate and reflect likely results from the appeals process, they cover only those situations in which an appeal is possible. For National Courses this means that no improvements are made within a grade, for instance from band 2 to band 1. The automatic procedure will not allow for improvements of more than one National Course grade or two Standard Grade grades. Changes of more than this magnitude will be based on examiner consideration of coursework evidence in the appeals procedure.

Standard Grade and National Courses differ in the structure of their results. The first qualification has a grade for each element, while the second has grades which are divided into bands. Standard Grade appeals are possible for any difference between estimated and achieved external grades, while National Course appeals require that the difference between the estimate and achieved award bands spans at least two grades. These differences have been translated into two different, but similar rules, one for each type of qualification.

3Standard Grade

The standard Grade rule operates on an element basis, as in the appeals process.

Rule

Standard Grade

The following rule is applied element by element within a subject.

Candidates achieving one or two grades below their estimate will be upgraded to this estimate if three conditions are met:

1there are at least 6 candidates with this estimate in the centre

2at least 60% of the centre’s candidates with the same estimate have an external result equal to or one grade better than the estimate

3the candidate has taken the examination papers to which the estimate relates

The procedure does not apply to candidates estimated at grade 7 or for whom no estimate has been submitted.

Explanation

Take for example a candidate who has an external result at grade 4, but was estimated at grade 2. Let’s assume that the centre has estimated 10 candidates in total at grade 2. If one candidate has an external result at grade 1 and five candidates have a grade 2, this centre meets the first two conditions. There are more than six candidates with an estimated grade 2, and 60% of these (six of the ten) have a result either at the estimated grade (2) or one better (grade 1). The candidate has also sat the Credit papers, so an award at grade 2 can be derived.

Conversely, if the candidate had achieved grade 7, it would not have been possible to derive an enhanced award, because the change from 7 to 2 is considered too high to be left to an automatic process. Note also, that if the candidate had not taken the Credit papers (and was not an authorised absentee), it would have been impossible to award a derived grade.

Rationale

Under the previous procedure only about a third of centres were concordant on the basis of sufficient agreement between estimates and external results over the whole range of grades. For this reason rules had been added which considered agreement on one or two grades only. In practice, these additional rules accounted for many derived grades. For this reason, considering agreement over two grades has now been taken as the main approach.

A minimum of 6 candidates is required to be certain that a good agreement is not due to chance. When requiring 60% agreement over a minimum of five candidates for example, 3 candidates on the estimate or one grade above, would meet the criterion, and this was considered to be unacceptable.

The inclusion of candidates achieving one grade above their estimate reflects a part of the previous procedure which was included to ensure that candidates in centres which consistently underestimate their candidates’ eventual attainment were not disadvantaged.

Sixty percent is considered to be the lowest acceptable level of agreement. This percentage takes into account that estimates and external results generally do differ to a certain extent, so that a higher percentage agreement might be unrealistic. On the other hand, it also recognises that a lower percentage agreement would not give a strong enough indication that the estimate is at the right grade.

Changes

The new procedure brings a number of changes. The following paragraphs deal with changes which are obvious or may have implications for centres.

One of the most prominent changes is that concordance factors will no longer be computed.

This factor was computed differently for severe and for other centres. However, a sufficient concordance factor was not a necessary requirement. If it did not meet the criterion, agreement was computed over one or two bands. The complicated computations and use of this factor have been replaced by a percentage computed for every estimated grade in Standard Grade. The percentage agreement will be computed in the same way for all centres, whether they are lenient or severe. Severity of estimates is dealt with through extending agreement to results one grade better than the estimate.

Another change is the restriction of upgrades to two grades maximum. Upgrades, from grade 5 or 6, or even from grade 7, to grade 2 and 1 are too large to be dealt with automatically. If the centre is convinced that these candidates have underperformed, appeals may be submitted. In the appeal Examiners will give due consideration to the evidence submitted.

The new procedure no longer awards any improvements to other candidates in a centre who had been ‘leapfrogged’. This happened when the derived grade for one candidate was better than the grade achieved by another candidate who had a better estimate. The reason to discontinue this is that the appeals procedure would never award improvements purely on this basis.

Reports

The ‘median report’ which was issued to each centre proved to be difficult to understand. One reason for this was that it reflected the situation before grades were derived, but was produced after grades had been derived. The information that a centre was both concordant and severe or both concordant and lenient was not helpful.

The new report simply lists the candidates whose grades have been derived, and gives a general indication of the difference between the centre’s estimates and external results (before these have been adjusted by the Derived Grades procedure).

Impact

All in all, and looking across all elements and subjects, the new procedure is expected to award about as many derived grades as before. There will be no upgrades for candidates whose result is more than two grades lower than their estimate, or to ‘leapfrogged’ candidates. More candidates who just missed their estimate will be upgraded.

4National Courses at Intermediate 1 and 2, Higher and Advanced Higher

For National Courses, the procedure mirrors the appeals procedure, which operates by grade. Bands are grouped into grades as follows:

BandGrade

1-2A

3-4B

5-6C

7CA (Compensatory Award)

8-9NA (No Award)

The estimated band will continue to be used in other procedures. For this reason it is essential that centres continue to estimate bands as best as they can. Estimation patterns will be monitored to ensure that action can be undertaken if changes occur.

Rule

The following rule is applied to each combination of subject and level, eg Mathematics Higher.

National Courses

Candidates one grade below their estimate will be upgraded to their estimated band if two conditions are met:

1there are at least 6 candidates with an estimated band at the same grade in the centre

2at least 60% of the centre’s candidates with estimated bands at the same grade have achieved this grade

Explanation

Take for example a candidate who has an external result at band 7 (often resulting in a compensatory award), but was estimated at band 6, grade C. Let’s assume that the centre had estimated four candidates at band 5 and six candidates at band 6, in total 10 candidates at grade C. If two candidates have an external result at band 5 and four candidates a result at band 6, this will meet the condition of 6 candidates at grade C. Our candidate’s, and all other candidates’ awards estimated at band 5 or 6 who had achieved an external result at grade CA (band 7), will be upgraded to their estimated band.

Rationale

Judging agreement by grade mirrors more closely the appeals procedure than the previous mixture of overall concordance and agreement in one or two bands.

In the past, higher criteria had been set at SCE Higher Grade than at Standard Grade because of the greater degree of externality and higher stakes of this exam. However, this did not take into account that the overall level of agreement between estimates and achieved external results was, and still is, lower than for Standard Grades, probably because estimates have to be made quite early in a shorter course. A 60% agreement is considered to be the lowest acceptable agreement. This percentage takes into account that estimates and external results generally do differ to a certain extent, so that a higher percentage agreement might be unrealistic. On the other hand, it also recognises that a lower percentage agreement would not give a strong enough indication that the estimate is at the right grade.

Changes

The new procedure brings a number of changes. The following paragraphs deal with changes which are obvious or may have implications for centres.

One of the most prominent changes is that concordance factors will no longer be computed.

However, a sufficient concordance factor was not a necessary requirement. If it did not meet the criterion, agreement was computed over one or two bands. The complicated computations and use of this factor have been replaced by a percentage computed for every estimated grade. The percentage agreement will be computed in the same way for all centres, whether they are lenient or severe. Severity of estimates is dealt with through extending agreement to external results over two bands within the same grade.

The new procedure does not award any improvements anymore to other candidates in a centre, who had been ‘leapfrogged’. This happened when the derived grade for one candidate was better than the grade achieved by another candidate who had a better estimate. The reason to discontinue this is that the appeals procedure would never award improvements purely on this basis.

The maximum upgrade was 2 bands. This has now been changed to the estimated band in the next grade up. This ranges from one band, eg from ‘grade’ CA at band 7 to grade C at band 6, to three bands, eg from grade C at band 6 to grade B at band 3.

Reports

The so-called ‘median report’ proved to be difficult to understand. One reason for this was that it reflected the situation before grades were derived, but was produced after grades had been derived. The information that a centre was both concordant and severe or both concordant and lenient was not helpful.

Impact

All in all, and looking across all subjects, the new procedure is expected to award a few more derived grades than before. There will be no upgrades for ‘leapfrogged’ candidates, but the maximum upgrade is more flexible.

For centres with small numbers of candidates it will continue to be necessary to appeal, as their numbers do not provide a firm enough basis for an automatic procedure.

5Appeals

In National Courses, it used to be possible to appeal for candidates who had received the maximum derived grade but were still short of their estimate. Under the new rules, however, a derived grade will be at the estimated band, or no grade will be derived at all. Therefore, there should be no need for appeals against the derived grades listed in the new report.

6Monitoring

The change to the new rules has been informed by their application to historical data and by comparisons with the results of the previous rules. The procedure will continue to be closely monitored to evaluate its effect on results, appeals and on estimating practice.

21/09/2018 RvK derived grades 2003 draft1.docPage 1 of 8

Appendix: the new external derived grades report

Scottish Qualifications Authority August 2004

National Qualifications 2004

Comparison of Estimates and External Grades and Derived Grades

(Comparison before derived grades were awarded)

Centre: 9999999 Scottish school

Standard Grade

Subject Code / Element Code / Subject / Element / Mean estimate as compared to mean external result / Derived Grades awarded to SCN
(other than absentees)
0010 / 0011 / Accounting and Finance / Knowledge & Understanding / Within same grade / 022654321
022264311
0010 / 0012 / Accounting and Finance / Handling Information / One grade lower / 022264311
0010 / 0013 / Accounting and Finance / Practical Abilities / DG not applicable / DG not applicable
National Courses

Subject code