The Coronavirus pandemic has forced us to confront the status quo in many walks for life, and there is no doubt that many things will no longer be the same even once it has passed, remote work among them.
The topic of this post is undeniably small fries given the breadth and severity of the current global problem, but let’s examine the impact of the new world order on something trifling: the administration of high stakes medical exams.
So, yeah–what about those computer-based multiple-choice exams that ABR forces radiologists to travel across the country and congregate in closed quarters for?
Please note that these arguments also largely apply to the NBME’s USMLE exams and every other board exam, but I’m focusing on the Core Exam here because the ABR has complete control over the process as both the creator and administrator of its exams, it’s a smaller and more manageable group of people involved, and because they unnecessarily require plane travel and a two-day hotel stay for the majority of examinees. (Also, even the supervillain NBME announced today that they are looking at options for alternate test delivery!)
The Current Problem
The ABR has announced it currently plans to administer the Core Exam at their Chicago and Tuscon centers in November. In addition to the old cost concerns and the new safety concerns, there’s also a simple practical concern: we have no idea if we’ll be in the midst of another shelter-in-place shutdown this November. This backup plan is just a backup hope right now. What will the ABR do if things fall apart, administer two sets next June? People are supposed to get hired for jobs or practice independently before the Core Exam is even graded?
Even with the new planned November test date, we will still almost certainly be living in a world where having a thousand high-risk doctors fly across the country for no reason, eat at the same buffet breakfast, check-in on the same handful of laptops, then share a testing room, snack area, and bathroom for two days should be an unacceptable option. And, unchanged from before, how can we stop burdening residents with additional expenses? Between the Core Exam and the Certifying Exam, a resident will need around three nights of hotel and two roundtrip flights, easily wasting $1 million for each cohort of residents that could be better spent on literally anything else.
No doubt, the Core Exam is not a great test. A closed-book MCQ knowledge assessment is a poor measure for minimum competency for safe independent practice. But, for better or worse, it is the measure we’re working with. So how can we administer this very portable exam to residents in a way that conforms to safety concerns in our new post-COVID world while also ensuring fair and valid result?
There are several ways the ABR could handle this. I’d like to see video-proctored at-home testing. But if the ABR won’t do that (and they should), my proposed solution: it’s time to disseminate the Core Exam to residency programs.
The Core Exam Can Already Technically Be Disseminated
It is a universally held opinion that it’s an expensive waste of money and time to force trainees to travel to Chicago or Tuscon for the two-day multiple-choice painfest that is the Core Exam. Ironically, despite the ABR’s stated inability to offload exam administration to local commercial testing centers like Prometric, they have instead already shown that it is entirely feasible to disseminate ABR content to any regular old PC in the world, which they demonstrated after the Mammogeddon Saga of 2017.
In that year, an unspecified bug prevented a large number of examinees in Chicago from receiving the mammography module during the test. The chosen “solution” at the time was then to have those examinees take the module remotely from wherever they wanted at a later date.
As an aside, you might be wondering how it was okay to have a portion of this high-stakes exam completely unproctored? Well, the fact is that the structure of the Core Exam makes performance on an individual module (including but not limited to mammography) except physics essentially irrelevant. The original grading scheme used for individual section performance consistently demonstrated that residents either performed sufficiently poorly across the board to fail or do well enough overall to pass. This is why the ABR stopped with the pretense that you could “condition” (i.e. fail) a single module (except for physics) back in 2018. The mammo module dissemination was just a pretense to check an awkwardly unchecked box.
Regardless of the underlying merits and psychometric significance of that debacle, what 2017 conclusively demonstrated is that it is technically feasible for ABR exam material to be taken outside of Chicago in Tucson, even if not at a Prometric center. The technological hurdles are manageable.
How to Disseminate the ABR Core Exam
To understand how plausible decentralized exams would be, it’s important to understand how the ABR manages its own testing centers, which may be unfamiliar ground for most diplomates familiar only with the oral boards.
To its credit, the ABR does not treat its test security like most commercial testing centers. Taking a test at a Prometric center is to subject yourself to something between the TSA screening at an airport and a prison cavity search. While the ABR states that all items must off your person, they do not force you to turn out your pockets, they do not wand you with a metal detector, and no one is patting anyone’s crouch for contraband. Bathroom breaks do not require signing in and out with your driver’s license, a live photo, and your signature; you just go to the bathroom. While they originally and creepily posted a staff member in the bathroom for the whole day during the first two years, staff now check the bathrooms on an intermittent roving basis. You can wear a sweatshirt; you can take off your sweatshirt.
There are staff members observing the testing room and the break area, and there are cameras (though it’s unclear if those are actively monitored or most likely there to investigate retroactively if irregularities occur). The room is brighter than a reading room but not as starkly lit as is common in most testing centers. Each computer has a conventional LCD monitor and a run of the mill PC.
This is all to say that the ABR’s on-site security policy is appropriate for evaluating a bunch of professionals and less like the one employed to ferret out potential criminals that most doctors are familiar with from the MCAT and USMLE. And the equipment requirements are trivial.
Given this precedent, it’s not too hard to imagine ways in which a program could reasonably administer the Core Exam if truly remote at-home testing is felt to be undesirable. A dedicated room or rooms with computers sufficiently spaced out with reasonably accessible bathroom and break space for snacks is the only physical space requirement. A webcam could be set up so that the test room could be streamed and recorded for the ABR’s benefit/review, and examinee position within the room would be relayed so the ABR knows who is who. An additional webcam could also be set up in a designated break room if necessary.
Program staff could proctor. Cities with multiple programs could also swap staff to proctor each other for extra fun. If the ABR is concerned about the trustworthiness of local personnel (particularly security outside of the monitored testing room), which is the most obvious exam security concern, then this would be an opportunity to employ the large number of ABR governors, trustees, and volunteers around the country who could potentially help proctor the exam. Travel of a single individual from outside an institution (likely nearly always from local or other drivable distance) would be far less disruptive than the mass travel required in the current system. Presumably one or both of the ABR centers could remain open for local or regional programs as well as those programs that for whatever reason are unable or unwilling to administer the exams themselves.
The solution isn’t interesting or complicated. It also wouldn’t be that hard. Or, hey, just do it at home with remote proctoring.
To be sure, my rapier wit does not make me an expert in testing administration. However, I am confident that any insurmountable obstacles to disseminating the Core Exam are political and not technological.
Other ABR Exams
Administration of other ABR exams, such as the Certifying and CAQ exams would be somewhat more complicated as the examinees in many cases would be spread throughout the community and no longer affiliated with a training program. I suspect many regional institutions would be very willing to host these additional exams, particularly if the ABR provided a small financial incentive. Heaven knows the ABR has funds aplenty for this purpose.
The current travel mandates are even more absurd for the one-day Certifying Exam or the half-day (at most) CAQ Exams. A strong alternative plan would be to cancel the Certifying Exam altogether on the basis of it being a redundant waste of time for all parties involved. The CAQ exams, instead of being taken over a year after completing fellowship should just be taken in June at the end of Fellowship at the candidate’s local program. Problem solved, forever.
The Home Version
Or, you could just take it at home.
This is certainly the safest option, and one that the ABR would likely reject out of hand despite being entirely workable. It’s also the obvious approach for many exams thanks to the virus.
If the ABR’s exam wasn’t so focused on knowledge over skill, it could be open book (like real life), and cheating as such would be largely irrelevant. However, in its current form, the ABR could still ensure a fair experience in an unproctored environment. The ABR software can log or block problematic keystrokes (e.g. copy, cut, task-switching, etc) while open, disable screen recording and screenshots, and require that the webcam and microphone are on the whole time to document that the right person is taking the exam with no funny business. In order to prevent cheating during bathroom breaks, the software could lock in all previous answers to any viewed questions prior to starting the break. If you’ve seen a question, you must answer it before taking a break, thus removing the incentive and ability to try to look up answers away from the camera’s prying eye.
Again, problem solved, forever.
What the ABR shouldn’t do
Try to continue business as usual. The world is not business as usual in any facet or function right now, and nationwide travel to take a computer-based exam is frankly unacceptable for at least 2020 if not substantially longer. We’re talking people possibly dying unnecessarily as a result.
The ABR, in its current state of purported transparency, has not described any plan or effort to do anything other than delay some exam dates. They need to start rethinking that policy now, because a remote solution is the inevitable and only defensible outcome.
4 Comments