The ABR and the Practicability of Doing Its Job

Yesterday, the American Board of Radiology announced in an email blast that it was canceling the planned in-person administrations of the Core and Certifying Exams and postponing until an unspecified time in 2021.

I wasn’t the only person to tell the ABR that its November in-person exam plan was–from its inception–magical thinking. So on one hand, the announcement was long overdue. On the other, they’re still missing the point: the only solutions to the problem of high-stakes medical exams right now are disseminated ones, and the number of other boards moving to virtual exams continues to grow steadily. Centralized nationwide travel is simply a nonstarter, and the delays here only serve to lengthen the duration of the ABR’s missed opportunity to do the right thing.

Unlike the June cancellation, this time the ABR gave no date. That’s because any date would be a meaningless guess. There is no end in sight for this pandemic. We have no idea when it will be appropriate for nationwide travel for something as stupid as a multiple-choice test. There is a highly nonzero probability that anything of the sort in 2021 will remain unconscionable. So while they caved to the reality that the November date was untenable, they won’t acknowledge the deeper reality that their testing centers became obsolete forever earlier this year.

The vague notion of a “2021” administration is a plan without a plan. Programs can’t meaningfully change rotation and call schedules for a second time to a random new date. Residents trying to account for these high-stakes exams in their family planning are once again left helpless. Instead, everyone will need to contend with a combination of two classes of residents taking the exam in rapid succession or even at the same time. This is a completely avoidable logistical disaster for training programs.

Remote Testing: The Obvious Solution

Here is what the ABR said about remote testing in their email:

In response to numerous queries, delivery of the [Core and Certifying Exams] using available remote platforms is not practicable. We have investigated many options, but the inability to adequately control image quality, the testing environment, and security would significantly threaten the fairness, reproducibility, validity, and reliability of the testing instrument across all candidates.  We want to give an exam that will accurately reflect the hard work of our candidates on their path to certification.

Candidates and trainee advocates deserve to know why. They deserve to know what those investigations were and why the ABR feels no solutions can be used. They deserve to know why if “available” options won’t work why the ABR will not create their own. After all, they created their own test and built their own testing centers! And then, when one of those centers failed in 2017, they even managed to administer part of that exam remotely to candidates at home! And finally, we all deserve to know why the ABR changed from using practical to practicable in its correspondence.

It’s completely insufficient to just say it’s not practicalpracticable to do a remote solution.

Take ownership of this communication.

Literally–and I mean literally–no one believes you. A simple “it’s not going to happen” is not a suitable justification for the continued disruption, inconvenience, cost, and health concerns of an in-person exam.

The current exam set up was suboptimal for years, but now it’s inappropriate.

To be fair, I believe the ABR when they say that commercial testing centers are not a viable option right now. Even the NBME has been forced to enroll medical schools to help administer the Step exams given issues with Prometric during the pandemic. These places are canceling exams left and right in order to maintain a semblance of social distancing, but even on a good day, these companies provide a poor examinee experience. It’s 2020 and we can do a good job with exam security without subjecting candidates to the degrading experience of the Prometric pat-down. (On a related note, this would be a great time for the ACR to roll out a program-administered in-service exam to demonstrate how to do this to the ABR).

Regardless, it’s possible to do live-proctoring via webcam. It’s possible to do live-proctoring in-person at residency programs. There are multiple options to administer an exam that may be harder than “flipping a switch” (their words) but are still solutions worth actively pursuing. I wrote about some of these common-sense approaches in April.

A Pathologically Absent Sense of Irony

The ABR says that remote testing cannot happen because it would prevent them from ensuring the “reproducibility, validity, and reliability of the testing instrument across all candidates.”

But these delays have already ruined this exact thing, and further delay will only do so more. While these exams are theoretically criterion-referenced by the Angoff committees, at the end of the day, the Core exam is an exam taken by rising senior residents in June after a set three-year period and an established curriculum. Every class is measured at the same time in the same place.

The ABR is focused on the physical setting, but they are ignoring the timing.

For the residency class of 2021, it will be an exam taken by senior residents deep into their final spring, potentially at the very end of their training (or yes, if we’re being honest, during fellowship). There will be residents who may have begun subspecializing via mini fellowships. For ESIR residents, they may be months from their last diagnostic rotation.

If the Core Exam really tested things that every radiologist should know, then perhaps these seasoned seniors would pass at rates higher than ever before. But as those who have taken the Core Exam already know, that isn’t the case. These seniors will need to study hard (again) to maintain the superfluous working-knowledge to succeed over the additional months of waiting time. When they sit down for the exam in Chicago or Tuscon, they will be exhausted in addition to bitter, angry, and maybe even scared. They will be wearing masks and trying to stifle errant coughs. It will be a miserable and unnecessary experience.

Between the delay and the atmosphere, this will never be the same instrument. But the beauty of the ABR’s inscrutable system is that their answer to any grading questions will always be trust us.

Endlessly repeating a presumably earnest desire for reproducibility, validity, and reliability is the best the ABR can muster, but it reminds me of recent Trump comments about mail-in ballots: tangentially related to a real concern but ultimately hysterical, impotent, and rooted in fear.

Follow the Money

The gut reaction of everyone is that the ABR won’t do remote testing because they want money. And I want to push back on that a little because it’s an oversimplification. It’s not just about money.

The ABR will take your fees regardless of how they run the exam. Yes, they’ve invested in developing and running their own testing centers, but those empty rooms gobble up money whether they’re being used or not. Some of their reluctance may have something to do with the duration and terms of their leases, but they aren’t sharing. Ultimately, that’s a sunk cost no matter what the future holds. While it would certainly cost some money to develop a way to share the exam widely (which they may have already done the heavy lifting for), they wouldn’t lose much money upfront if exams were taken at residency programs or at home this year.

The big problem with a remote test is that it’s a long term one-way street. The ABR has maintained for years that it alone could administer these exams. But if we can have a remote exam now, then that position is simply not true, which means they’ve either been wrong for all those years, or worse, lying. I don’t think we should discount their reticence to own up to past mistakes. You don’t have to watch the news often to know that doubling down against an inconvenient truth is common practice. This was a strategic error from the day it was announced.

If the ABR does eventually use a commercial center, for example, then yes, some of those fees will go to the center, and the ABR will lose some profits. For example, this random Prometric document says they charge $105 for 8-hours of seat time (to be clear I have no idea if this is accurate). A trainee will pay 5 years of $640 fees to take the Core and Certifying Exams, so currently $3200. If we estimate that the three total days of exam time (2 Core, 1 Certifying) would cost in the neighborhood of $315, then just under 10% of your total fees would be lost. This is what sacrificing your health and convenience is worth to the ABR.

Can the ABR deal with that financial hit? Well, yes, of course. They are a multiple-choice question development company primarily staffed by volunteers with $50 million in the bank.

But if the ABR were to be true pioneers here and permanently move to program- or home-based testing, then there would be little financial hit at all.

A Sad Reality

Ultimately this is about optics and control, not about practicalities practicabilities. The ABR–entrenched in a decade-old decision to manage their own testing centers and now doubling down on their no-remote party line during the pandemic–is just doing the unequivocally wrong thing.

It seems that $50 million in cash reserves is insufficient financial lubrication to help them move past their prior strategic missteps because make no mistake: this is truly a psychological–not practicable–barrier. Shame is a powerful motivator for cognitive dissonance.

Every resident, fellow, and advocate should engage with every single organized radiology society to sign a unified letter of concern. It may not matter, but the ABR should know they stand alone.

 

4 Comments

Eddie 06.04.20 Reply

We feel your pain here in radiation oncology. Oral boards (typically taken at the end of first year as attending), originally scheduled for May 2020, were postponed to October 2020 and now to an undisclosed date in 2021. These poor first-year attendings have been prepping in anticipation for an entire year and are now stuck in limbo.

Rad onc leadership organizations, such as ASTRO (national society), ARRO (resident society), and SCAROP (society of academic chairs) have sent letters to the ABR. I encourage our radiology colleagues to do the same.

ASTRO Letter: https://www.scribd.com/document/464396547/ASTRO-Letter-to-the-ABR

ARRO Letter: https://www.scribd.com/document/464396553/ARRO-Letter-to-the-ABR

SCAROP Letter: https://www.scribd.com/document/464396261/SCAROP-Letter-to-the-ABR

Positron 06.04.20 Reply

Spot on, as usual.

Dual boarded in Radiology/Nuclear Medicine. Had the opportunity to CAQ in Nukes, glad I didn’t, went the ABNM route, which was a Pearson test center visit. The ABR could do the same. They *choose* not to. In fact…I did one recert @ RSNA. Then they decided it wan’t good for 10 years and changed the rules. Funny how they could do it offsite then, huh?

The ABMS/ABIM probably started this entire mess decades ago to address their bad financial position. Until the ACR fills (PLEASE) the void or legal action forces change (or I retire) I/we are stuck playing *THEIR* game.

If the ABMS/ABR would have offered CME for a reasonable price (consisting of MOC type questions, with feedback related to how your peers did) rather than trying to cram this down our throats things could have been different.

Anthony Mancuso,MD 06.07.20 Reply

We have a solution developed at the University of Florida over the last 10 years. The validity of this authentic competency-based evaluation rubric is supported by the 2 part paper referenced at the end of this note.

We have delivered this to 1300 residents in about 45 US ACGME programs. We have a method of registration and a method of interacting with proctors that is all on line and well established and well accepted by all parties which has been administered by essentially 2 people over the last 10 years.

Would be simple to scale that up and use testing centers at medical schools throughout the United States at nominal cost.

We have a proven grading rubric that is extremely effective in that we can train people to grade these examinations which are inherently subjective in the responses by the test takers since they are essentially looking at the study and writing down what they would write down in a report or verbal consultation to a referring doctor, but in fact can be approached with a grading key making reliability across graders highly reliable as is proven by our 2 papers. This approach is what makes the evaluation rubric authentic. The individuals are looking at all available DICOM images and doing what they would normally do without the bias or games inherent in a multiple-choice format no matter how the psychometricians believe they can create an authentic test with such a methodology.

The test is more authentic than the prior oral exam which was far better than current multiple-choice approach that cannot possibly measure competency either peer to peer or in general. The oral exam was hampered by the lack of inclusion of negative cases, which are included in our evaluation rubric, and to some extent hampered by the subjectiveness of the evaluators who in my experience attempted to be very fair, although inherently not as objective as our system might provide.

The expense of traveling examiners to that dingy motel in Louisville and disruption of the very value physicians schedules for a week can be essentially totally avoided even while paying a per graded case fee to subspecialty domain graders. Graders can be trained relatively rapidly and confidently and the grading rubric is being constantly upgraded and improved.

Beyond that this authentic testing methodology can generate data showing common errors in interpretation and educational gaps which is a extraordinarily important contribution to radiology with regard to targeting future uses of AI as well as areas of curriculum emphasis.

If the ABR is not want to adopt this proven authentic certification rubric we have made it known to the ACR that they certainly could become an alternative certifying authority at far less expense to the ACR and to those who must pay the expenses for traveling to test centers to take a certifying examination that clearly could be delivered in a secure and honest manner by creating a network of medical school proctors and testing centers who am sure would not charge excessively for their human or physical resources.

None of this is new information to some individuals in the ACR.

We at UF are here to help. I truly believe no one else anywhere has the type of data or establishment of a proven authentic competency-based evaluation rubric with the features I’ve just described.

I have made attempts to discuss this with ABR leadership but even with high-level back channel encouragement to engage in such a discussion, I have not had a positive response to date.
https://www.academicradiology.org/article/S1076-6332(20)30167-7/pdf

https://www.academicradiology.org/article/S1076-6332(20)30169-0/pdf

The team here at UF is here to help if asked.

Leave a Reply