30 October 2020
 
Addressed to:
Head of Examinations, The Royal College of Surgeons of England
Head of Examinations, The Royal College of Surgeons of Edinburgh
Head of Examinations, The Royal College of Physicians & Surgeons of Glasgow
Head of Examinations, The Royal College of Surgeons in Ireland

------------------------------------------------------------------------------------------------
 
We, the underscored signatories, write with reference to the recently held Intercollegiate MRCS A examination conducted 15 September 2020.

This year has presented many changes, challenges and uncertainties for junior doctors. COVID-19 has changed many of the ways in which we work and live. Healthcare workers have dealt with this unprecedented change in their stride, with people uniting across the world to face up to the challenge of looking after our patients to the best of our ability.

As a global cohort of surgical trainees, we show full appreciation that every new system may encounter natural teething issues and difficulties in delivery. However, we believe that this should not be at the expense of trainees’ professional development and personal wellbeing. As aspiring surgeons, we look to the Royal Colleges for support and setting an example of excellence for our future workforce.

Unfortunately, we feel that the Royal Colleges have jointly failed in their delivery of the MRCS A examination and list our formal grievances as follows:

1.   Poor pre-examination communication:
·      Once told that the examination was to be delivered online - instructions were expected “No later than 4 weeks prior to examination date” (email sent 17/07/2020). Partial instructions were eventually received on 25th August (3 weeks before examination). These instructions were updated on the online link in the email weeks later, without any notification. Final instructions were not sent until 4 days prior to exam.This was the first notification we received regarding the change in timings of papers.
·      The mock examination itself consisted of just 5 questions. The examination screen layout was different in each mock, so this created confusion as to what would be available on the day. 

2.   Poor Delivery and commencement of examination:
·      For many, the password to access the examination was sent just 5 minutes before the exam start time (when login advised 30minutes before). This email also contained guidance information, giving the candidates only minutes to review these. This stress could potentially have seriously affected performance in the first examination paper.
·      We received no instructions to begin the examination – but a “Do Not Proceed Until Instructed” page.
·       Numerous candidates were not able to access papers as expected and have in turn, been required to repeat the examination either in November or January.
·       Candidates with disabilities have expressed concerns regarding the nature of the examination. All candidates with dyslexia were offered extra time, as is standard examination procedure. However, there are concerns relating to the requirements of the online sitting not taking into account candidates with disabilities. One candidate with ADHD has expressed that “Sitting in one spot for up to two hours without moving one’s head was extremely challenging”. We echo their concerns in this letter.
Examples of difficulties encountered on the day of the examination as submitted by trainees:

“I was unable to upload the second half of my first paper for ~1 hr. Tried to speak to examsoft on their online chat and couldn't get through to an advisor for 45 minutes, also rang their helpline and was charged £63 because I was on hold for half an hour. Also emailed RCS exam support at the time and never got a response. Logged out to log back in to try again then couldn't log in for ages 
Part of me wishes I'd chosen to resit now!”
 
“ I wasn’t able to access my paper 2 on 15/9/20 despite being able to complete and upload both parts of paper 1 with no issue. It kept saying the server failed and then eventually wouldn’t let me in. I tried getting through to examsoft and couldn’t get hold of anyone - I was no 10 in the line...
I eventually managed to get hold of someone at MRCS support department who was getting advice from examsoft. Essentially I was in the phone for 2 hours and got no where - I was essentially told i couldn’t access it despite restarting my computer etc and that I wasn’t the only one and to await further instruction and a later date”
 
“1. Unable to have access to the software and the mock exams, the week before the exam, this required several emails and calls to the college and Examplify team before it was corrected. 
2. Password to access exam not made available till 9:09am, when exam was due to start at 9:15am.
3. Several screen freezes throughout exam, when it worked skipped several questions, had to go back forth etc.
4. Took >30/40 minutes to upload afternoon paper and several reload attempts.”
 
3. Issues with uploading examinations:
·       There are numerous reports of candidates not being able to access papers. Indeed, while we were told to turn off all electronic communications as part of the examination rules, the college proceeded to send email communications dates 15 September 2020 at 12:56, noting:

"We have been alerted by ExamSoft that they have detected an issue preventing some exam takers accessing the ICBSE exams, which they are actively and urgently working to resolve. Depending on the amount of time it takes solve this issue, a corresponding amount of additional time maybe added to the exam to account for the delay”.

·       Candidates could easily have missed this email due to following examination rules stringently, or indeed still be attempting to take their examination.
·       Many candidates were only able to partially sit their exams. Others faced issue with uploading the marks or video of their examination.
·       A number of candidates have been instructed that they had used a feature that crossed out answer choices rather than select them. This highlights the lack on instruction relating to the software. These candidates were only informed about this in late October, not allowing them a reasonable chance of sitting the examination on 3rdNovember.
·       Some candidates have been informed their marks have simply “disappeared” and that the college has no result of their performance. Despite the college’s effort, this has clearly demonstrated an inability to treat all candidates fairly and equally.

4. Poor post-examination communication and postponement of results:
·       An overwhelming number of candidates note issues in their response to email queries sent following exam, regarding check-in process / uploading of results.
·       Results are usually 3-4 weeks post exam as stated on the Royal College website. Results for this examination were amended to the expected date of 23rd October 12:00. Notification of delayed results was sent at 11:27 (33 minutes before expected time) owing to “unforeseen circumstances”. This caused many trainees a lot of anxiety. Results were then delayed until 28th October 12:00.
·       On the 28thOctober, it was not possible to log in to RCS Eng website to attain results and some candidates were still trying to access results more than 24 hours after the release, causing undue stress to those individuals.
·       There were no emails of apology and no communication as to when to expect results. The first form of communication from the college (RCS Eng) was a “tweet”, excluding all those who do not subscribe to this social media platform or use it frequently.
·      The notification on RCSEng website that results by email not possible “due to confidentiality” was incongruous, as RCSEd were able to deliver results this way. This shows a clear difference in the treatment of candidates between colleges, despite regulations clearly stating that candidates will not be disadvantaged between examination centres/bodies due to the intercollegiate nature of the examination.
·      This also has led to a reduced preparation window for those wishing/needing to re-sit the examination in January.
·      A number of candidates have been informed that their answers were not registered by the software and they were only informed on 25/10/20 not allowing them the opportunity to sit the examination again in November.
·       A number of candidates have not been responded to regarding the resit examination on November 3rdtherefore denying them the opportunity.

5. Failure of result delivery
·      Despite being assured of result delivery, multiple individuals are still not in receipt of their marks. There are individuals who have not received their mark today due to discrepancy in their ExamSoft upload. With the numerous factors already mentioned with this system, it seems highly likely that there are errors with this system.
·      Candidates have reported notifications of malpractice for not being visible in the examination screen despite successful ID checks. They have subsequently been denied examination results without being provided evidence of this. They were not alerted to this during the examination.
 
6. Examsoft Detection of Malpractice and College Procedure Regarding Suspicion of Malpractice
·      The College has informed candidates at irregular intervals of suspected malpractice regarding their Conduct in the MRCS examination. Despite assurances of a new date for final results, multiple candidates have been sent an email 22.10.20 with accusations of malpractice. Reasons for this included lack of visibility in the screen. These candidates are extremely anxious about the allegations of malpractice and seeking advice from the relevant regulatory bodies.
·      The college has now failed to align their Malpractice policy, with different policies available online showing discrepancy. Most pertinent is the 2 versions of section 3.1;

Live Version of Malpractice Procedure Link from RCSEdinburgh Website 29.10.20: https://www.rcsed.ac.uk/exams/exam-policies-and-procedures

“3.1: Allegations of malpractice may be reported to the College by assessors, examiners, invigilators, candidates, examinations staff, patients or simulated patients, or examination venue staff. When dealing with alleged malpractice, the College will deal both with the invigilator or supervising examiner and the candidate or the candidate’s representative.”
 
Live Version of Malpractice Procedure Link From ISCP Website 29.10.20:
https://www.intercollegiatemrcsexams.org.uk/mrcs/mrcs-regulations-and-formal-documents/

“3.1. Allegations of malpractice may be reported to Colleges by Assessors, examiners, invigilators, remote proctoring services, candidates, examinations staff, patients or simulated patients, or examination venue staff. When dealing with alleged malpractice, Colleges will deal both with the invigilator, remote proctoring service or supervising examinerand the candidate or the candidate’s representative.”

This difference in policy calls in to question the very validity of being able to call in to question candidate’s suspected malpractice via video link.We are of the opinion that it is completely inappropriate to have different versions of policy live via web link as candidates will have no idea which to follow - especially when there has no communication regarding updated policy via email or otherwise.

7.   Disproportionate increase in pass mark:
·       Previous published pass marks as per ICBSE reports are 71.9%, 71.5%, 70.3%, 70.6%, 70.9%, 69.5% (Average 70.73%).The pass mark for this sitting raised to 76.9%.
·       There has been no transparent communication relating to this to date and the communication so far does not indicate whether this takes into account that there were two cohorts sitting examination – April and September. Therefore, passing numbers expected should be proportional to the number of candidates in a standard sitting.
·       Many candidates have been studying since beginning of year, so a higher mark may well be expected, but the September cohort should not be penalised for additional study time afforded to those who were ready to sit in April but examination delayed. Not to mention those doctors who had been working very hard in redeployed roles during the pandemic without additional opportunity or mental space for revision.
·       This rise in pass mark also disregards the fact that anyone was given the opportunity to resit, without providing any evidence of technological issues during the exam. Given that those who performed well were unlikely to give up their results, it follows that those that chose this option had a lower average score. Removing them from the score analysis further skews the results.
·      Looking at other Royal Colleges the average pass mark for Part 1 (Part A equivalent) is: MRCP 58-62% MRCPCH 62-65% MRCOG 65% MRCGP 67-72% (Av 68%) Whilst RCS needs to uphold certain standards, the 76.9% pass mark disregards the clear level of knowledge held by all those who have scored well, despite the new format and uncertainty surrounded by this examination. 

Sought Outcomes From Complaint:
The below sought outcomes seek to rectify the unfair treatment of candidates across the board. We expect the College should look to enact the following so that all candidates undertaking the exam are treated fairly and equitably:

1.    Transparent evidence of the pass mark setting process, pass rate and evidence that the pass mark was not artificially inflated to “cap” numbers passing as the examination should be about a competence and knowledge level achieved, not seeking to “limit” those who have attained the relevant knowledge. This would include, but not be limited to, evidence that this analysis took into account:
a) The fact that two cohorts (April 2020 and September 2020) were sitting this examination, therefore an increased number of candidates would be expected to pass
b) The inclusion or exclusion of marks of those who completed both papers but opted to resit in November, or the skew caused by elimination of these results a
c) The possibility that cheating has gone undetected
d) The standard setting process of the “minimum expected competency” from the expert panel, used in calculating the cut off mark of pass / fail
2.    A review of the use of ExamSoft for conducting future examinations of the MRCS A
3.    Re-evaluation of the algorithm used to set pass mark – to ensure such an unanticipated rise in pass mark does not occur again and trainees can plan their revision accordingly. 
4.    For “extenuating circumstance” to apply this exam sitting and to not count as one of the 6 attempts allowed before being barred from undertaking the MRCS A examination
5.    A formal apology regarding the poor communication, delivery and for the stress caused by multiple delays in release of results.
 

We look forward to your response.
 
Yours sincerely,
 
Aspiring surgeons.