The standard for all MRCPCH theory exams is set using an absolute criterion; as opposed to a norm-reference approach, there is no number or percentage of candidate predetermined to pass or fail.
Hypothetically speaking, using a criterion-referenced approach all candidates could pass the exam or fail the exam. As a result of the standard setting process, the pass mark and pass rate may vary for each diet, however the difficulty is considered when setting the pass mark.
The specific type of standard setting used in the RCPCH is a Modified Angoff Method. Following each exam diet a panel of professional judges who are subject matter experts grade the difficulty of the exam, by item, which results in a cut-score or pass mark.
Prior to grading the exam, judges are asked to conceptualise the notion of the 'borderline candidate'. The borderline candidate is a fictitious person who has a 50% chance of passing the exam. Once this has been established, the judges are asked to independently grade each item of the exam and rate out of 100 borderline candidates the percentage of how many will answer the question correctly. Once completed these rating are compiled and prepared for a panel discussion.
The judges will then come together as a panel to examine the ratings for each item in sequence and discuss each item’s rating. Judges will discuss disparity in ratings and different viewpoints. Once these have been considered, if they wish, judges will have an opportunity to re-rate items based on the discussions, taking into account:
- the judgements of the entire panel
- the definition of the borderline candidate
- the comments of the panel.
Once these judgments have been cast, the candidates’ performance in the actual exam is then revealed, via P+ values and biserals, and the judges give final ratings. These judgements are the last ratings provided, which when averaged become the final criterion referenced score for an item. The weighted mean of all these final scores is calculated and the nearest achievable mark becomes the pass mark for the exam.
At the beginning of the panel discussion, a Standard Monitoring Meeting (SMM) is also held, where any anomalous item will also be identified and discussed for suitability. If it is decided that the items were unsuitable for exposure, they will be removed and not contribute to the overall pass mark.