Little known fact, but the ABR recently added a public member to the Board of Governors. She recently wrote a hello article for The BEAM.

How boards like the ABR certify that their members have the requisite skills and knowledge to benefit patients is not well understood by the public, nor by many people in health care. I am learning about the hard and important work done by the ABR, largely behind the scenes and not apparent to the ultimate beneficiaries, the American people.

It’s also not well understood by the ABR. How exactly does the ABR certify radiologist skill again?

When I teach MBA students who are interested in health care about how organizations like the ABR ensure quality, they are surprised. They know more about the roles that government and private insurance companies play in what practitioners can and can’t do. They find it reassuring that groups like the ABR operate solely in the public’s interest. They are impressed that even as medical knowledge has explosively expanded, specialty boards have continued to meet their missions.

Solely in the public’s interest? Not quite.

It’s true that the public-facing mission of the ABR is “to certify that our diplomates demonstrate the requisite knowledge, skill, and understanding of their disciplines to the benefit of patients.”

But I guess no one told the new governor and first public member of the board that the ABR, by its own bylaws, acts in the ABR’s best interest.

Section 4.6. Conflicts of Interest. It is the policy of this Corporation that the legal duty of loyalty owed to this Corporation by a Governor serving on the Board of Governors of this Corporation requires the Governor to act in the best interests of this Corporation, even if discharging that duty requires the Governor to support actions that might be contrary to the views, interests, policies, or actions of another organization of which the Governor is a member, or to the discipline of which the Governor is a member. Consistent with a Governor’s duty of loyalty, a person serving as a Governor of this Corporation does not serve or act as the “representative” of any other organization, and his or her “constituency” as a Governor of this Corporation is solely this Corporation and is not any other organization or its members.

Did she not have to sign some sort of absurd legalese-filled contract confirming her unwavering loyalty to the ABR? Because I was asked to just to keep taking my OLA questions last week.

The Coronavirus pandemic has forced us to confront the status quo in many walks for life, and there is no doubt that many things will no longer be the same even once it has passed, remote work among them.

The topic of this post is undeniably small fries given the breadth and severity of the current global problem, but let’s examine the impact of the new world order on something trifling: the administration of high stakes medical exams.

So, yeah—what about those computer-based multiple-choice exams that ABR forces radiologists to travel across the country and congregate in closed quarters for?

Please note that these arguments also largely apply to the NBME’s USMLE exams and every other board exam, but I’m focusing on the Core Exam here because the ABR has complete control over the process as both the creator and administrator of its exams, it’s a smaller and more manageable group of people involved, and because they unnecessarily require plane travel and a two-day hotel stay for the majority of examinees. (Also, even the supervillain NBME announced today that they are looking at options for alternate test delivery!)

The Current Problem

The ABR has announced it currently plans to administer the Core Exam at their Chicago and Tucson centers in November. In addition to the old cost concerns and the new safety concerns, there’s also a simple practical concern: we have no idea if we’ll be in the midst of another shelter-in-place shutdown this November. This backup plan is just a backup hope right now. What will the ABR do if things fall apart, administer two sets next June? People are supposed to get hired for jobs or practice independently before the Core Exam is even graded?

Even with the new planned November test date, we will still almost certainly be living in a world where having a thousand high-risk doctors fly across the country for no reason, eat at the same buffet breakfast, check-in on the same handful of laptops, then share a testing room, snack area, and bathroom for two days should be an unacceptable option. And, unchanged from before, how can we stop burdening residents with additional expenses? Between the Core Exam and the Certifying Exam, a resident will need around three nights of hotel and two roundtrip flights, easily wasting $1 million for each cohort of residents that could be better spent on literally anything else.

No doubt, the Core Exam is not a great test. A closed-book MCQ knowledge assessment is a poor measure for minimum competency for safe independent practice. But, for better or worse, it is the measure we’re working with. So how can we administer this very portable exam to residents in a way that conforms to safety concerns in our new post-COVID world while also ensuring fair and valid result?

There are several ways the ABR could handle this. I’d like to see video-proctored at-home testing. But if the ABR won’t do that (and they should), my proposed solution: it’s time to disseminate the Core Exam to residency programs.

The Core Exam Can Already Technically Be Disseminated

It is a universally held opinion that it’s an expensive waste of money and time to force trainees to travel to Chicago or Tuscon for the two-day multiple-choice painfest that is the Core Exam. Ironically, despite the ABR’s stated inability to offload exam administration to local commercial testing centers like Prometric, they have instead already shown that it is entirely feasible to disseminate ABR content to any regular old PC in the world, which they demonstrated after the Mammogeddon Saga of 2017.

In that year, an unspecified bug prevented a large number of examinees in Chicago from receiving the mammography module during the test. The chosen “solution” at the time was then to have those examinees take the module remotely from wherever they wanted at a later date.

As an aside, you might be wondering how it was okay to have a portion of this high-stakes exam completely unproctored? Well, the fact is that the structure of the Core Exam makes performance on an individual module (including but not limited to mammography) except physics essentially irrelevant. The original grading scheme used for individual section performance consistently demonstrated that residents either performed sufficiently poorly across the board to fail or do well enough overall to pass. This is why the ABR stopped with the pretense that you could “condition” (i.e. fail) a single module (except for physics) back in 2018. The mammo module dissemination was just a pretense to check an awkwardly unchecked box.

Regardless of the underlying merits and psychometric significance of that debacle, what 2017 conclusively demonstrated is that it is technically feasible for ABR exam material to be taken outside of Chicago in Tucson, even if not at a Prometric center. The technological hurdles are manageable.

How to Disseminate the ABR Core Exam

To understand how plausible decentralized exams would be, it’s important to understand how the ABR manages its own testing centers, which may be unfamiliar ground for most diplomates familiar only with the oral boards.

To its credit, the ABR does not treat its test security like most commercial testing centers. Taking a test at a Prometric center is to subject yourself to something between the TSA screening at an airport and a prison cavity search. While the ABR states that all items must off your person, they do not force you to turn out your pockets, they do not wand you with a metal detector, and no one is patting anyone’s crouch for contraband. Bathroom breaks do not require signing in and out with your driver’s license, a live photo, and your signature; you just go to the bathroom. While they originally and creepily posted a staff member in the bathroom for the whole day during the first two years, staff now check the bathrooms on an intermittent roving basis. You can wear a sweatshirt; you can take off your sweatshirt.

There are staff members observing the testing room and the break area, and there are cameras (though it’s unclear if those are actively monitored or most likely there to investigate retroactively if irregularities occur). The room is brighter than a reading room but not as starkly lit as is common in most testing centers. Each computer has a conventional LCD monitor and a run of the mill PC.

This is all to say that the ABR’s on-site security policy is appropriate for evaluating a bunch of professionals and less like the one employed to ferret out potential criminals that most doctors are familiar with from the MCAT and USMLE. And the equipment requirements are trivial.

Given this precedent, it’s not too hard to imagine ways in which a program could reasonably administer the Core Exam if truly remote at-home testing is felt to be undesirable. A dedicated room or rooms with computers sufficiently spaced out with reasonably accessible bathroom and break space for snacks is the only physical space requirement. A webcam could be set up so that the test room could be streamed and recorded for the ABR’s benefit/review, and examinee position within the room would be relayed so the ABR knows who is who. An additional webcam could also be set up in a designated break room if necessary.

Program staff could proctor. Cities with multiple programs could also swap staff to proctor each other for extra fun. If the ABR is concerned about the trustworthiness of local personnel (particularly security outside of the monitored testing room), which is the most obvious exam security concern, then this would be an opportunity to employ the large number of ABR governors, trustees, and volunteers around the country who could potentially help proctor the exam. Travel of a single individual from outside an institution (likely nearly always from local or other drivable distance) would be far less disruptive than the mass travel required in the current system. Presumably one or both of the ABR centers could remain open for local or regional programs as well as those programs that for whatever reason are unable or unwilling to administer the exams themselves.

The solution isn’t interesting or complicated. It also wouldn’t be that hard. Or, hey, just do it at home with remote proctoring.

To be sure, my rapier wit does not make me an expert in testing administration. However, I am confident that any insurmountable obstacles to disseminating the Core Exam are political and not technological.

Other ABR Exams

Administration of other ABR exams, such as the Certifying and CAQ exams would be somewhat more complicated as the examinees in many cases would be spread throughout the community and no longer affiliated with a training program. I suspect many regional institutions would be very willing to host these additional exams, particularly if the ABR provided a small financial incentive. Heaven knows the ABR has funds aplenty for this purpose.

The current travel mandates are even more absurd for the one-day Certifying Exam or the half-day (at most) CAQ Exams. A strong alternative plan would be to cancel the Certifying Exam altogether on the basis of it being a redundant waste of time for all parties involved. The CAQ exams, instead of being taken over a year after completing fellowship should just be taken in June at the end of Fellowship at the candidate’s local program. Problem solved, forever.

The Home Version

Or, you could just take it at home.

This is certainly the safest option, and one that the ABR would likely reject out of hand despite being entirely workable. It’s also the obvious approach for many exams thanks to the virus.

If the ABR’s exam wasn’t so focused on knowledge over skill, it could be open book (like real life), and cheating as such would be largely irrelevant. However, in its current form, the ABR could still ensure a fair experience in an unproctored environment. The ABR software can log or block problematic keystrokes (e.g. copy, cut, task-switching, etc) while open, disable screen recording and screenshots, and require that the webcam and microphone are on the whole time to document that the right person is taking the exam with no funny business. In order to prevent cheating during bathroom breaks, the software could lock in all previous answers to any viewed questions prior to starting the break. If you’ve seen a question, you must answer it before taking a break, thus removing the incentive and ability to try to look up answers away from the camera’s prying eye.

Again, problem solved, forever.

What the ABR shouldn’t do

Try to continue business as usual. The world is not business as usual in any facet or function right now, and nationwide travel to take a computer-based exam is frankly unacceptable for at least 2020 if not substantially longer. We’re talking people possibly dying unnecessarily as a result.

The ABR, in its current state of purported transparency, has not described any plan or effort to do anything other than delay some exam dates. They need to start rethinking that policy now, because a remote solution is the inevitable and only defensible outcome.

Update: Two days after this post was written, the ABR announced they were essentially following the recommendations at the bottom of this post including removing all the problematic language from the agreement and extending a new less onerous version to all diplomates, including those who had already signed the version I’ve outlined below. I have not yet seen this new version, but the ABR states it now just focuses on outlining the terms of MOC and not stealing the ABR’s copyrighted intellectual property. If this pans out as promised, it would be by far the most responsive the ABR has been to any stakeholder concern for at least the past decade.

You’ve signed dozens if not hundreds of EULAs over the years. Those are the “end-user licensing agreements” that pop up whenever you install software on your computer or start using a new service. They’re usually filled with pages of legalese, and no one reads them.

Presumably as a response to last year’s lawsuit, the ABR has foisted a mandatory draconian agreement on all of its candidates and diplomates. The difference between the usual EULA and this is that a real EULA is a choice.

You should read this one—even when you inevitably sign it—because it’s yet another stellar example of the ABR’s heavy-handed tone-deaf approach to just about everything within its tiny little purview.

The two versions (one for “candidates and diplomates” and one for MOC) are almost identical, so we’ll just break down the best parts of the MOC agreement.

Findings

By entering into this Agreement for Diplomates (hereinafter the “Agreement”), I pledge myself to the highest ethical standards in the practice of my discipline.

So far so good.

I agree to disqualification from examination or from renewal of a certificate in the event that any of the statements herein made by me are false, or if I violate any of the policies, rules and regulations, or the Bylaws of the Board.

You’re the one really making statements for me in this, but okay, I get it. On a related note, everyone should really read the ABR bylaws.

I recognize the trustees of the Board as the sole and only judge of my qualifications to receive and to retain a certificate issued by the Board. I understand and agree that as a diplomate of the Board, I have the responsibility to supply the Board with information adequate for the Board’s proper evaluation of my character and my credentials.

I take the first sentence to mean that we promise not to try to have any other competing board or entity do MOC, as it has recently been argued in court that initial certification and maintenance of certification are actually separate products and should be untied, freeing other entities such as the NBPAS to provide competing MOC products. Signing this is essentially saying you agree with the court that the ABR’s monopoly is totally okay.

Additionally, I hereby request and authorize any hospital or medical or professional organization of which I am a member, have been a member, or to which I have applied for membership, and any person who may have information which is deemed by the Board to be material to its evaluation of my registration or certification, to provide such information to representatives of the Board upon their request. I agree that communication of any nature made to the Board regarding my registration or certification may be made in confidence and shall not be made available to me under any circumstances. I hereby release from liability any hospital, medical staff, medical or professional organization or person, and the Board and its trustees and other representatives, from liability for acts performed in good faith and without malice in connection with the provision, collection, or evaluation of information or opinions, whether or not requested or solicited by the Board in connection with my registration or certification.

You agree that anyone you have ever or will ever meet should spy on you for the board’s pleasure, and the board will never tell you who. Despite dramatic verbiage, this is actually how state medical boards and lots of other certifying organizations function. That said, it’s not hard to imagine a world where the ABR goes digging for dirt on people it doesn’t like.

I understand and agree that in consideration of my registration, my moral, ethical and professional standing will be reviewed and assessed by the Board; that the Board may make inquiry of the persons named in my registration form and of such other persons and entities as the Board deems appropriate with respect to my moral, ethical and professional standing; that if information is received which would adversely affect my registration, I will be so advised and given an opportunity to rebut such allegations, but I will not be advised as to the identity of any individual or entity who has furnished adverse information concerning me; and that all statements and other information furnished to the Board in connection with such inquiry shall be confidential, and not subject to examination by me or by anyone acting on my behalf.

Due process, presumably. But I will point out that people are scared of the ABR. One anonymous complaint and your livelihood could be dragged in the mud. What’s the threshold when it comes to moral and professional behavior? Is mocking the ABR on Twitter still okay?

I agree that I will not use any litigation process, subpoena or other means to cause or attempt to cause any disclosure of the contents of any registration form, including my own, or any proceedings of any committee’s evaluation of such registration form or of my certification, whether such disclosure is by operation of law or otherwise.

What about litigation to end a monopoly? Oh, that’s a separate paragraph.

This one is a little confusing at first but seems to suggest that any deliberation of the ABR behind closed doors will always be confidential. You agree, for example, that even if the ABR were to opaquely and perhaps unfairly take away your certification, that you have no legal means to find out the real reason why, fight back, or obtain recompense.

I accept that the Board determines admissibility to all MOC examinations, and that each examination is supervised by proctors who are responsible to the Board and are empowered by the Board to ensure that the examination is conducted ethically and in accordance with the rules of the Board. I understand that I must bring government-issued photo identification to any examination that I attend. Such government-issued photo identification includes one of the following: state-issued driver’s license, military ID, passport, or state-issued ID.

This is a little odd because most diplomates now have to do OLA and no longer take MOC exams, but a similar passage is in the agreement for residents/fellows for the initial certification exams. This does suggest that any dissemination of the ABR’s exams (a topic I will be returning to in the near future) would require an amendment to this agreement.

I agree that the Board is not liable for information provided to the medical community or to the public regarding my certification status, and I further agree that I will promptly notify the Board of any error or omissions in such information.

It’s not the ABR’s fault for making mistakes in its core functions.

I understand and agree that the continued validity of my certificate will be contingent upon my meeting the requirements of the Maintenance of Certification Program (ABR-MOC) administered by the Board, as amended from time to time. I understand that the ABR-MOC program is designed to monitor my professional standing, lifelong learning and self-assessment, cognitive expertise, and practice quality improvement, each an MOC component for which I am responsible.

This is a tough one because stating that I “understand” that MOC is “designed” to monitor “cognitive expertise” is a false statement, which we agreed not to do at the beginning of the agreement. We all know that MOC in its current form is revenue generation thinly veiled in trivial box-checking.

I agree to participate in ABR-MOC in accordance with and subject to stated policies, rules and regulations, as amended from time to time, including timely payment of fees. The Board does not undertake any responsibility to provide individual diplomates with notice of changes to MOC policies. I further understand it is my responsibility to stay informed regarding all aspects of the MOC program and my progress therein, through my personal database and the ABR website. I will keep truthful and accurate records of my participation in the MOC program, and I will promptly advise the Board of any change of my current contact information.

The ABR of interviews and “the Beam” newsletter and the ABR of constant defensive legal posturing are not the same organization. I will quote ABR president Brent Wagner: “One of the fundamentals I’ve been encouraging is to take ownership of flawed or incomplete communications.”

So, it’s the ABR’s responsibility to do that, except not really, because we want to be able to blame you if we do a bad job.

I understand that it may be necessary to revise and update this Agreement at a later date, and that as a condition of continued certification and/or participation in MOC, that I may be required to execute and return to the Board a revised Agreement, which shall replace and supersede the terms of this Agreement.

Necessary and required. Gives you the warm fuzzies.

I waive and release and shall indemnify and hold harmless the Board and its trustees, directors, members, officers, committee members, employees, and agents from, against and with respect to any and all claims, losses, costs, expenses, damages, and judgments (including reasonable attorneys fees) alleged to have arisen from, out of, with respect to or in connection with any action which they, or any of them, take or fail to take as a result of or in connection with this Agreement, any examination conducted by the Board which I apply to take or take, the grade or grades given me on the examination and, if applicable, the failure of the Board to issue me a certificate or qualification or the Board’s revocation, suspension or probation of any certificate or qualification previously issued to me and/or the Board’s notification of any interested parties of its actions.

As in, I promise I will never sue the ABR no matter what.

That is one extremely long almost unreadable sentence (with 21 commas!), but, of course, that’s the point.

Impression

In summary, this is pathetic.

As a non-lawyer, I’m not even sure this kind of nonsense is legally enforceable. One could at least make the argument that these contracts are signed under duress (e.g. excessive economic pressure) and are thus null and void. “Voluntarily” signing a non-negotiable contract in order to be board-certified, which is a functional requirement to practice, certainly seems like at least a gray area to me.

I am under the impression, however, that duress defenses for contract breaches are not trivial to prove, and this agreement is certainly intended first and foremost as a way to discourage any further lawsuits.

While there is new fierce opposition within the radiology community about these agreements, they are actually largely unchanged dating back to at least 2013 for candidates (with the main addition that we’re also now also waiving our FERPA rights). It would appear the main change is spreading the love to diplomates. However, that the ABR conveniently waited until after the most recent amended complaint was filed in the recent lawsuit before rolling out this iteration is likely no coincidence. I look forward to seeing how much the ABR spent on legal fees in 2019 that resulted in them having their lawyers cook this up.

It’s easy to forget when dealing with the ABR that board certification is supposed to be a form of physician self-governance. Doesn’t feel that way, does it? Reading all this, it’s easy to lose sight of the fact that this organization’s primary function is converting multiple choice questions written by volunteers into money.

As I mentioned in a recent post, ABR president and soon to be highly-compensated Executive Director Brent Wagner has described his goal to increase communication and transparency. If he means that, and I have no reason to believe otherwise, then this is a perfect opportunity to follow through on that promise. Unilaterally dropping a compulsory heavy-handed one-sided agreement upon which every radiologist’s ability to practice in their field is predicated is exactly the kind of opaque aloofness that the ABR has been promising it doesn’t want to employ anymore.

The ABR has plenty of staff and several dedicated executives, including a head of external relations with a six-figure salary. From a communications perspective, these are unnecessary and objectively bad choices of the ABR’s own making and the organizational equivalent of scoring on your own goal.

Here’s what the ABR should do

1. Say mea culpa for completely blowing, as per usual, a PR opportunity by simply dropping this bullshit in everyone’s myABR account without warning and then immediately breaking the OLA site functionality to coerce you to sign it without explanation or discussion.
2. Flush this steaming turd back where it came from, including for those who already signed it, and then create a new reasonable agreement, preferably with stakeholder involvement. Even if this “agreement” is largely unchanged from years past, it’s still garbage, and it says a lot about the current level of trust in the ABR that people are this disgusted by a reiteration of what is traditionally a meaningless form.

At the very least, they should do with sincerity what I’ve done sarcastically above, translating paragraph by paragraph the obtuse legalese into plain English and then providing the rationale and background for imposing such one-sided legal powerplays.

It’s not that I can’t guess. It’s that I shouldn’t have to.

Radiographics hosted a Twitter chat this week to discuss a recent op-ed, “Have We Done Radiology Trainees a Disservice by Eliminating the Oral Board Examination.” I was asked to participate.

ABR President and soon to be highly-compensated Executive Director Brent Wagner was also set to throw down and managed to fire off a couple of tweets at the beginning before disappearing. He answered the first of five pre-shared discussion questions and only responded to one direct question. You can read the full thread collected here if you’re interested.

The first question: “Does the ABR Core Exam test radiology competence?”

#RGchat T1: the ABR certification exam is intended to test knowledge as it relates to competence, and critical thinking as it relates to image interpretation. Other elements of competence (procedures, professionalism, etc) are better tested (assessed) by the residency faculty.

— Brent Wagner (@brentwagner99) April 14, 2020

That sounds like doublespeak to me, but I think this is probably as straightforward an answer as the ABR can provide. It also makes the unfortunate admission that we are essentially testing for a simile.

Per the ABR, its mission is “to certify that our diplomates demonstrate the requisite knowledge, skill, and understanding of their disciplines to the benefit of patients.” We are testing for #1, but we can and should also be testing directly for #2.

The Core Exam is a knowledge assessment, and knowledge assessments based on multiple-choice questions are reductive and intellectually lazy. Knowledge in isolation is likely one of the least significant measurables in determining if a radiology trainee is safe to practice for the “benefit of patients,” especially in a world with easy access to electronic resources. So despite purporting to assess for skill in its mission statement, we are testing for knowledge and pretending that the correlation between knowledge and competence is so high that it can stand alone as the sole determinant of minimal competence.

We could, however, directly test for competence and critical thinking by designing a test where the diagnostic portion simulates actual radiology practice and not an artificial multiple-choice single-best-answer format.

#RGchat T1: Much of the exam seeks to assess a candidate's ability to choose the most likely diagnosis based on a set of images. Similar to what I expect of myself and my colleagues in daily work.

— Brent Wagner (@brentwagner99) April 14, 2020

Leaving aside the other parts of the exam that are irrelevant to practice, the issue that Wagner sidesteps (and that made up a large fraction of the discussion on Twitter) is that choosing a single likely diagnosis from a list is an unnecessary artificial construct being used for psychometric convenience. An MCQ test is cheap to create, easy to administer, and easy to validate. The ABR once said it was creating “the test of the future,” but it really just replaced two smaller MCQ tests and an oral exam with one longer MCQ test. It was only the “future” in the sense that it was announced before it happened. I don’t get primed by answer choices in real life, and that’s the difference between knowing the correct answer and merely recognizing it.

Today, a recent interview with Wagner was included in this Radiology Business article:

One of the fundamentals I’ve been encouraging is to take ownership of flawed or incomplete communications. In other words, if an ABR stakeholder doesn’t understand something, that’s our problem, not theirs, and we have the responsibility to do whatever it takes to fix it.”

So, fix it.