In response to concerns about exam administration, the American Board of Radiology has now released a “Statement on Delivering Remote Exams” to their Coronavirus updates page:

In response to queries, delivering our exams remotely is problematic. We have investigated many options, but the inability to adequately control image quality, the testing environment, and security would significantly threaten the fairness, reproducibility, validity, and reliability of the testing instrument across candidates.

A challenge? Sure. Problematic? A pure cop-out. To be absolutely clear, the ABR has already used remote testing in the past. It was just on their terms due to their mistake. There is nothing about ABR exam administration that cannot be effectively recreated in a remote setting, especially under these unprecedented circumstances.

We want to give exams that fairly and accurately assess a candidate’s knowledge and experience.

We’ve long since moved past wants and desires here. Nobody wants to be dealing with this, but forcing nationwide travel is by far the greatest threat to fair and accurate candidate assessment.

We pledge to remain flexible and responsive to candidate needs, and we appreciate everyone’s patience as we all go through the pandemic’s effects together.

Being “responsive to candidate needs” and also rejecting the idea of remote testing are 100% incompatible. This is absolutely a public health issue, and a head-down blinders-on approach is unacceptable for a health-related organization purporting to act in the public’s interest.

Last week, people were upset about the heavy-handy liberty-destroying verbiage of the ABR’s new MOC agreement. Some people complained. At least one cranky fellow wrote a blog post about it, and then two days later the ABR backtracked and promised to remove the language about waiving various rights.

They did, in fact, do that. But, oddly, they didn’t release the full text on their website as people requested (and certainly not a track changes-type comparison or an explanation). Those who already signed the old agreement thus can’t see the new agreement for which they are automatically being “grandfathered.”

(It’s perhaps also worth noting that the ABR ironically used the term grandfathering incorrectly. Grandfathering is when old people get to keep using old rules despite subsequent changes. The ABR does this literally every day when lifetime certificate holders are exempt from MOC. Automatically making a new agreement retroactive is the opposite of grandfathering.)

Regardless, here is the new agreement (with commentary, natch).

I UNDERSTAND AND AGREE that by enrolling in the American Board of Radiology (ABR) Maintenance of Certification (MOC) Program, I am subject to the policies and procedures of the program and agree to hold myself to the highest ethical standards in the practice of my specialty.

Deal.

I UNDERSTAND AND AGREE that the ABR MOC program is designed to monitor my professionalism and professional standing, lifelong learning and self-assessment, assessment of knowledge, judgement [sic] and skills and improvement in medial [sic] practice.

Yes, they really misspelled judgment and medical.

I UNDERSTAND AND AGREE that as a diplomate of the Board, I have the responsibility to supply the Board with information adequate for the Board’s proper evaluation of my credentials, moral, ethical and professional standing. I authorize any hospital or other medical or professional organization, or person who may have information relevant to its evaluation of my certification and/or enrollment in the MOC program to provide such information upon request. IF requested by the Board, I will sign and promptly return appropriate consents for release of information addressed to specific persons or entities.

This is a less scary wording of what amounts to the same material. As in, you will cooperate with ABR investigations and aid the ABR in information gathering about you when your privacy rights prevent the ABR from doing so without your consent.

They no longer mention things like waiving FERPA but presumably, all of the details still apply here. If you agree to let them do whatever they want, then you will be signing away those same rights during an investigation if requested. While this sounds bad, it’s hard to imagine the ABR being able to revoke certification from bad actors without this information unless they always wait for state boards and hospitals to act first and then just follow suit.

I UNDERSTAND AND AGREE that any communications made to the Board regarding my certification and/or enrollment in the MOC program may be made in confidence and will not be made available to me under any circumstances. I hereby release from liability any hospital, medical staff, medical or professional organization, person, or the Board of Governors from liability for acts performed in good faith and without malice in connection with the Board’s request for information.

This is also essentially the same. We can’t try to punish whistleblowers.

I UNDERSTAND AND AGREE that should information be received which would adversely affect my certification and/or enrollment in the MOC program, I will be so advised and given an opportunity to rebut such allegations. However, I will not be advised as to the identity of any individual or entity who has furnished adverse information concerning me, and that all statements and other information furnished to the Board in connection with such inquiry shall be confidential, and not subject to examination by me or anyone acting on my behalf. I also agree that I will not use any litigation process, subpoena or other means to attempt to cause any disclosure of adverse information received by the ABR regarding my character or certification.

Whistleblowers can maintain anonymity. I know this was troubling to many people but doesn’t actually bother me that much with the exception of theoretical reports that are not made in good faith. It’s common practice for good faith reports to be protected from liability. But malicious and false reports, once discovered, are typically not immune, and if the motivation behind a report were to become suspect, I’d like to see that language put in. It’s also hard to imagine being able to robustly rebut allegations if “all statements and other information” are confidential.

I CONSENT to have my name and demographic data, including type and date of all ABR certifications and maintenance of certification status included in any list or directory in which the names of diplomates of the specialty boards are published.

Uncontroversial and unchanged.

I AGREE the Board is not liable for any incorrect information provided to the medical community or to the public regarding the status of my certificates, and I further agree that I will promptly notify the board of any errors or omissions in such information.

CYA, meaning unchanged.

I UNDERSTAND AND AGREE that the continued validity of my certificate will be contingent upon my meeting the requirements of the Maintenance of Certification Program (MOC) administered by the Board, as amended from time to time.

I.e, unless the ABR one day loses a lawsuit, you will always need to pay ABR’s MOC tax to remain board-certified.

I UNDERSTAND AND AGREE it is my responsibility to stay informed regarding all aspects of the MOC program and understand that the Board does not have any responsibility to provide individual diplomates with notice of changes to MOC policies.

This is also largely unchanged and is a petty cop-out. The idea is presumably to prevent diplomates from pleading ignorance when they don’t fulfill some component.

I UNDERSTAND AND AGREE that it is my responsibility to notify the ABR of any changes in my mailing address, phone number and email address. All changes made by me via the website shall be accepted as legally binding, and will become the property of the ABR.

ABR, you can’t just seize my property just because I enter my address in your system…

I’m kidding, more or less. What this actually means is that the ABR can legally sell your contact info to the highest bidder (gross).

I UNDERSTAND AND AGREE that it may be necessary for the ABR to revise and update this Agreement at a later date, and that I may be required to sign the updated agreement, which will replace and supersede the terms of this agreement.

We can have an internet discussion again when that happens.

Summary

This is shorter and more readable. All the truly heinous stuff has been removed. Is it perfect? Of course not. Is the effect in some cases the same despite the softer language? You bet. Is it much better than it was last week? Undeniably.

Why the ABR is burying their fix is beyond me. They should, in fact, delete all the recent agreements and ask all diplomates to sign this new version. Everyone deserves to see what they’re signing, and the other agreements (even if “grandfathered”) should not continue to exist.

Little known fact, but the ABR recently added a public member to the Board of Governors. She recently wrote a hello article for The BEAM.

How boards like the ABR certify that their members have the requisite skills and knowledge to benefit patients is not well understood by the public, nor by many people in health care. I am learning about the hard and important work done by the ABR, largely behind the scenes and not apparent to the ultimate beneficiaries, the American people.

It’s also not well understood by the ABR. How exactly does the ABR certify radiologist skill again?

When I teach MBA students who are interested in health care about how organizations like the ABR ensure quality, they are surprised. They know more about the roles that government and private insurance companies play in what practitioners can and can’t do. They find it reassuring that groups like the ABR operate solely in the public’s interest. They are impressed that even as medical knowledge has explosively expanded, specialty boards have continued to meet their missions.

Solely in the public’s interest? Not quite.

It’s true that the public-facing mission of the ABR is “to certify that our diplomates demonstrate the requisite knowledge, skill, and understanding of their disciplines to the benefit of patients.”

But I guess no one told the new governor and first public member of the board that the ABR, by its own bylaws, acts in the ABR’s best interest.

Section 4.6. Conflicts of Interest. It is the policy of this Corporation that the legal duty of loyalty owed to this Corporation by a Governor serving on the Board of Governors of this Corporation requires the Governor to act in the best interests of this Corporation, even if discharging that duty requires the Governor to support actions that might be contrary to the views, interests, policies, or actions of another organization of which the Governor is a member, or to the discipline of which the Governor is a member. Consistent with a Governor’s duty of loyalty, a person serving as a Governor of this Corporation does not serve or act as the “representative” of any other organization, and his or her “constituency” as a Governor of this Corporation is solely this Corporation and is not any other organization or its members.

Did she not have to sign some sort of absurd legalese-filled contract confirming her unwavering loyalty to the ABR? Because I was asked to just to keep taking my OLA questions last week.

The Coronavirus pandemic has forced us to confront the status quo in many walks for life, and there is no doubt that many things will no longer be the same even once it has passed, remote work among them.

The topic of this post is undeniably small fries given the breadth and severity of the current global problem, but let’s examine the impact of the new world order on something trifling: the administration of high stakes medical exams.

So, yeah—what about those computer-based multiple-choice exams that ABR forces radiologists to travel across the country and congregate in closed quarters for?

Please note that these arguments also largely apply to the NBME’s USMLE exams and every other board exam, but I’m focusing on the Core Exam here because the ABR has complete control over the process as both the creator and administrator of its exams, it’s a smaller and more manageable group of people involved, and because they unnecessarily require plane travel and a two-day hotel stay for the majority of examinees. (Also, even the supervillain NBME announced today that they are looking at options for alternate test delivery!)

The Current Problem

The ABR has announced it currently plans to administer the Core Exam at their Chicago and Tucson centers in November. In addition to the old cost concerns and the new safety concerns, there’s also a simple practical concern: we have no idea if we’ll be in the midst of another shelter-in-place shutdown this November. This backup plan is just a backup hope right now. What will the ABR do if things fall apart, administer two sets next June? People are supposed to get hired for jobs or practice independently before the Core Exam is even graded?

Even with the new planned November test date, we will still almost certainly be living in a world where having a thousand high-risk doctors fly across the country for no reason, eat at the same buffet breakfast, check-in on the same handful of laptops, then share a testing room, snack area, and bathroom for two days should be an unacceptable option. And, unchanged from before, how can we stop burdening residents with additional expenses? Between the Core Exam and the Certifying Exam, a resident will need around three nights of hotel and two roundtrip flights, easily wasting $1 million for each cohort of residents that could be better spent on literally anything else.

No doubt, the Core Exam is not a great test. A closed-book MCQ knowledge assessment is a poor measure for minimum competency for safe independent practice. But, for better or worse, it is the measure we’re working with. So how can we administer this very portable exam to residents in a way that conforms to safety concerns in our new post-COVID world while also ensuring fair and valid result?

There are several ways the ABR could handle this. I’d like to see video-proctored at-home testing. But if the ABR won’t do that (and they should), my proposed solution: it’s time to disseminate the Core Exam to residency programs.

The Core Exam Can Already Technically Be Disseminated

It is a universally held opinion that it’s an expensive waste of money and time to force trainees to travel to Chicago or Tuscon for the two-day multiple-choice painfest that is the Core Exam. Ironically, despite the ABR’s stated inability to offload exam administration to local commercial testing centers like Prometric, they have instead already shown that it is entirely feasible to disseminate ABR content to any regular old PC in the world, which they demonstrated after the Mammogeddon Saga of 2017.

In that year, an unspecified bug prevented a large number of examinees in Chicago from receiving the mammography module during the test. The chosen “solution” at the time was then to have those examinees take the module remotely from wherever they wanted at a later date.

As an aside, you might be wondering how it was okay to have a portion of this high-stakes exam completely unproctored? Well, the fact is that the structure of the Core Exam makes performance on an individual module (including but not limited to mammography) except physics essentially irrelevant. The original grading scheme used for individual section performance consistently demonstrated that residents either performed sufficiently poorly across the board to fail or do well enough overall to pass. This is why the ABR stopped with the pretense that you could “condition” (i.e. fail) a single module (except for physics) back in 2018. The mammo module dissemination was just a pretense to check an awkwardly unchecked box.

Regardless of the underlying merits and psychometric significance of that debacle, what 2017 conclusively demonstrated is that it is technically feasible for ABR exam material to be taken outside of Chicago in Tucson, even if not at a Prometric center. The technological hurdles are manageable.

How to Disseminate the ABR Core Exam

To understand how plausible decentralized exams would be, it’s important to understand how the ABR manages its own testing centers, which may be unfamiliar ground for most diplomates familiar only with the oral boards.

To its credit, the ABR does not treat its test security like most commercial testing centers. Taking a test at a Prometric center is to subject yourself to something between the TSA screening at an airport and a prison cavity search. While the ABR states that all items must off your person, they do not force you to turn out your pockets, they do not wand you with a metal detector, and no one is patting anyone’s crouch for contraband. Bathroom breaks do not require signing in and out with your driver’s license, a live photo, and your signature; you just go to the bathroom. While they originally and creepily posted a staff member in the bathroom for the whole day during the first two years, staff now check the bathrooms on an intermittent roving basis. You can wear a sweatshirt; you can take off your sweatshirt.

There are staff members observing the testing room and the break area, and there are cameras (though it’s unclear if those are actively monitored or most likely there to investigate retroactively if irregularities occur). The room is brighter than a reading room but not as starkly lit as is common in most testing centers. Each computer has a conventional LCD monitor and a run of the mill PC.

This is all to say that the ABR’s on-site security policy is appropriate for evaluating a bunch of professionals and less like the one employed to ferret out potential criminals that most doctors are familiar with from the MCAT and USMLE. And the equipment requirements are trivial.

Given this precedent, it’s not too hard to imagine ways in which a program could reasonably administer the Core Exam if truly remote at-home testing is felt to be undesirable. A dedicated room or rooms with computers sufficiently spaced out with reasonably accessible bathroom and break space for snacks is the only physical space requirement. A webcam could be set up so that the test room could be streamed and recorded for the ABR’s benefit/review, and examinee position within the room would be relayed so the ABR knows who is who. An additional webcam could also be set up in a designated break room if necessary.

Program staff could proctor. Cities with multiple programs could also swap staff to proctor each other for extra fun. If the ABR is concerned about the trustworthiness of local personnel (particularly security outside of the monitored testing room), which is the most obvious exam security concern, then this would be an opportunity to employ the large number of ABR governors, trustees, and volunteers around the country who could potentially help proctor the exam. Travel of a single individual from outside an institution (likely nearly always from local or other drivable distance) would be far less disruptive than the mass travel required in the current system. Presumably one or both of the ABR centers could remain open for local or regional programs as well as those programs that for whatever reason are unable or unwilling to administer the exams themselves.

The solution isn’t interesting or complicated. It also wouldn’t be that hard. Or, hey, just do it at home with remote proctoring.

To be sure, my rapier wit does not make me an expert in testing administration. However, I am confident that any insurmountable obstacles to disseminating the Core Exam are political and not technological.

Other ABR Exams

Administration of other ABR exams, such as the Certifying and CAQ exams would be somewhat more complicated as the examinees in many cases would be spread throughout the community and no longer affiliated with a training program. I suspect many regional institutions would be very willing to host these additional exams, particularly if the ABR provided a small financial incentive. Heaven knows the ABR has funds aplenty for this purpose.

The current travel mandates are even more absurd for the one-day Certifying Exam or the half-day (at most) CAQ Exams. A strong alternative plan would be to cancel the Certifying Exam altogether on the basis of it being a redundant waste of time for all parties involved. The CAQ exams, instead of being taken over a year after completing fellowship should just be taken in June at the end of Fellowship at the candidate’s local program. Problem solved, forever.

The Home Version

Or, you could just take it at home.

This is certainly the safest option, and one that the ABR would likely reject out of hand despite being entirely workable. It’s also the obvious approach for many exams thanks to the virus.

If the ABR’s exam wasn’t so focused on knowledge over skill, it could be open book (like real life), and cheating as such would be largely irrelevant. However, in its current form, the ABR could still ensure a fair experience in an unproctored environment. The ABR software can log or block problematic keystrokes (e.g. copy, cut, task-switching, etc) while open, disable screen recording and screenshots, and require that the webcam and microphone are on the whole time to document that the right person is taking the exam with no funny business. In order to prevent cheating during bathroom breaks, the software could lock in all previous answers to any viewed questions prior to starting the break. If you’ve seen a question, you must answer it before taking a break, thus removing the incentive and ability to try to look up answers away from the camera’s prying eye.

Again, problem solved, forever.

What the ABR shouldn’t do

Try to continue business as usual. The world is not business as usual in any facet or function right now, and nationwide travel to take a computer-based exam is frankly unacceptable for at least 2020 if not substantially longer. We’re talking people possibly dying unnecessarily as a result.

The ABR, in its current state of purported transparency, has not described any plan or effort to do anything other than delay some exam dates. They need to start rethinking that policy now, because a remote solution is the inevitable and only defensible outcome.