Section 1 Standard Setting Process

Section 1 Standard Setting Process

19th July 2013, Edinburgh

Introduction:

I was lucky enough to be given the opportunity to attend the Standard Setting Process on behalf of SURG. It was a very interesting and informative day. Approximately 15 consultant urologists gathered at The Royal College of Surgeons of Edinburgh to discuss and scrutinise the July 2013 section 1 examination of the FRCS Urol. The consultants were all members of the panel of examiners and/or panel of question writers.

The meeting began with a recap of the aims of the examination. These were then put into context in relation to the theory of Bloom’s taxonomy. The chairman asked the examiners to consider what passing would mean – would candidates be capable to practice independently, to operate competently, could they supervise a junior colleague and run a clinic? Most poignantly the chairman asked the examiners to consider whether they would be happy for the candidate to look after one of their family.

Discussion of anomalous questions:

The examiners first considered the anomalous questions; these are questions that statistically did not “perform well”. The goal was to either accept the question or to remove it from the examination. There are 3 flags:

1. A question that is too easy and therefore does not discriminate between candidates. The question is left in the paper but is sent back to the writing groups.
2. A negative discriminator – the “bad” candidates answer the question correctly but the “good” candidates do not. If less than 50% of candidates answer the question correctly then the question is removed.
3. A question that is answered differently to what would be expected (ie: is the answer in the bank incorrect?)

The questions discussed had ambiguous wording and an incorrect answer.

Angoff Procedure:

This is a procedure used to determine the pass mark for an examination used to certify or license practitioners. A candidate’s pass/fail performance is established independently of the group who sat the examination, and candidates are thus judged by comparing their performance to an absolute standard rather than other candidates. Theoretically, all candidates could pass.

The examiners were asked to consider a borderline (“just passing”) candidate and mark each question in the paper as how likely/probable that candidate would be to give a correct answer between 1 and 10. The marks are then averaged for each question. The sum of the averages creates a standard for each paper.

A borderline candidate was described as “a trainee who would just scrape an ARCP or one that would leave you feeling uncomfortable after a telephone conversation regarding your patient – even though they eventually would manage the patient correctly”.

To provide the eligibility to proceed mark an Angoff process is conducted.

The purpose of the standard setting process is to make sure that the doctor can demonstrate competence and that no incompetent candidate can pass.

To ensure precision around the pass mark the GMC recommends the 1 Standard Error of Measurement (SEM) is added to any standard set paper to ensure that no incompetent candidate can pass. For this examination the standard was set at 61.29%. The SEM was calculated as 2.89 providing an eligibility to proceed mark of 64.18%.

A KR20 (Kuder-Richardson 20) reliability factor is also measured for each examination based on the raw pass mark and the actual results. A reliable examination would discriminate well between good performing candidates and the poorly performing ones. The Urology Section papers are very reliable consistently gaining a score above 0.93 (when anything over 0.7 could be classed as reliable).

Section2:

Although the meeting was primarily to discuss section1 the examiners also touched on a few aspects of the section part of the examination. It was suggested that the vivas could concentrate more on higher order thinking and that better quality images should be used (using a tablet). Several scenarios were also discussed. The aim was to establish what a candidate should know and demonstrate to discriminate them from a 5 to a 6 (ie: pass/fail).

Conclusion:

The examination pass mark setting process is time consuming but is extremely fair and rigorous. The process is constantly evolving so feedback is incredibly important – candidates please always fill out the evaluation form!!

Angela Birnie

SURG Representative