In recent years, the American Board of Radiology (ABR) has utilized a membership fee model, where–for example–those working towards an initial certification in diagnostic radiology would pay a $640 annual fee until passing the Certifying Exam. Since one takes the Certifying Exam 15 months after finishing residency, that has meant recent diplomates have paid a specialty tax of around 1% of their gross income for a total of five years before enjoying the privilege of paying a mere $340 per year for MOC forever.

The fee schedule looked like this:

To illustrate, here’s my payment history (the annual fee actually increased a bit during my training because money).

As of September 2021, ABR has moved to “an exam fee model.” How does that look? Well, a one-time $640 application fee followed by a $1280 Core Exam fee and a $1280 Certifying Exam fee.

It doesn’t require a doctorate to note that the total cost for initial certification is the same: $3200.

That fee continues to put radiology in the highest echelon of medical specialties in terms of board costs, as enumerated in this 2017 paper (which incidentally undercounted the radiology costs).

What has changed is that this fee structure is now standard across other exams and is resulting in a decrease in the (otherwise ludicrous) subspecialty exam fees.

You see, until now, the much shorter half-day CAQ exams actually cost the most! As above, you can see I paid $3,280 this spring for the privilege of spending a morning taking a poorly formulated exam to pseudo-prove that I can totally do the thing I already do every day. That’s more than the cost of the combined total of the much, much bigger Core and Certifying Exams.

But, as of this September 17, 2021 update, it’s merely the same $640 application fee + $1280 exam fee for a total of $1,920 (a savings of $1,360!).

Of course, before you get any warm fuzzies about their generosity, keep in mind that the CAQ exams comprise a relatively small proportion of ABR revenues since only ~200 people take them every year, and, meanwhile, MOC revenues continue to grow year after year. The ABR, per its internal narrative and official documents, has recently been operating at a loss.

Thankfully, they have some retained earnings on hand to mitigate the red.

The February 2021 issue of the BEAM features a short article with the title, “Board, Staff Working Together to Control Expenses.”

As the Board of Governors discussed these new [remote] exam tools, one of the perceived potential benefits was the intuitive opportunity to decrease costs and, by extension, reduce fees. However, there are persistent barriers to fee abatement at the time of this writing, including the absence of proven success of the new exam structure; a lack of dependable forecasts of the future steady-state expense structure; the inherent long-term nature of established financial obligations related to exam center equipment and leases; and the unexpected short-term development costs of the virtual exam platform software.

Proven success? Check.

Does any stakeholder believe that ~$50 million in cash reserves isn’t enough to deal with “a lack of dependable forecasts of the future steady-state expense structure” [sic]?

They continue (emphasis mine):

ABR senior leadership is committed to working with the board to control costs. We are optimistic that this is achievable as we close in on the “new normal,” but we don’t know the extent of potential cost reductions, nor when they might be achieved. The less visible infrastructure elements of board functions, ranging from cybersecurity to volunteer support, are critical to customer service for our candidates and diplomates, as well as fulfillment of our core mission. Despite these obstacles, the board members view themselves as responsible stewards of ABR resources, both financial and otherwise. In this vein, they consistently challenge each other, and the ABR staff, to reduce costs and, subsequently, fees, to the extent possible.

Transparency, transparency, transparency. Anything less is just self-love.

Last year as the pandemic spiraled out of control, the ABR resisted–as they have for years–calls for disseminated exams away from their centers in Chicago and Tuscon. The lack of a foreseeable endpoint and pressure from advocates was finally enough for the ABR to make the switch. And to their credit, when the ABR came around, they went all the way: all exams are to be virtual from this point forward.

And guess what? It worked.

Apparently, it worked really well.

And I, for one, am not surprised.

People I’ve spoken to were overall very pleased with the remote experience. Were there rare technical difficulties? Sure. But reports are that the ABR was generally responsive and helpful in aiding candidates when issues cropped up, and multiple residents I spoke to gave ABR customer service high marks.

So while perhaps they shouldn’t have needed the worst pandemic in a century to make these changes, credit where it’s due: the ABR successfully pulled off the transition to at-home testing.

The ABR’s testing centers, though physically inconvenient, were always pretty nice compared with most commercial centers. But the ability to take the exam from a location of your choosing with no travel required and your choice of preferred snacks, clothing, thermostat settings, and bathroom is pretty nice. Having the exam over three days also probably helped with test-fatigue.

Future Fix Requests

There were a few complaints the ABR should address in future administrations:

• Answer choice strikethrough. This was a common request, and it’s a common feature including one available on the USMLE exams that residents are all used to.
• Cine clip optimization. This has been a longstanding complaint, but in this case, at least sometimes clips are presented in a separate window from the question and answer choices. They should be embedded the same way as normal images with easily controllable playback speed and the ability to manually scroll.
• Remove the 30 question auto-lock. The need to lock previously seen questions makes perfect sense at the end of a 60-question block and whenever a candidate takes an optional break. But I’m not sure I buy any justification for auto-locking mid-section. This is a true functional change from prior exam administrations that has a negative impact on those who would like to review all related questions before moving on. It’s also difficult to know how much time to allot to question review when you break up 120 questions into 4 blocks instead of two, making time management more difficult.
• Announce the section order. This was a big complaint I heard and one I agree completely with. For years the ABR has avoided publicizing the section order (e.g. Breast, then Cardiac, then GI) despite keeping it consistent across testing administrations. While people obviously aren’t supposed to discuss the exam, in the real world this has meant that candidates taking the exam later always know what sections are coming on which days, allowing them to cram most effectively. Unless every candidate has a randomized order, keeping this information semi-hidden in this setting just isn’t appropriate and should be a no-brainer to fix. Knowing you’re going to have ridiculous radioisotope safety microdetails on a specific day means you can prepare for that much more effectively and seriously jeopardizes the exam integrity. Again, this is not a new issue.

The Core Exam is still the Core Exam

Ultimately, the biggest complaint–no surprise–wasn’t the software but the test itself. It’s not as though the content magically became more on-point just because you got to wear pajamas.

If I were to limit myself to one content suggestion, it’s this:

I feel very strongly that the ABR’s reduction of physics and radiation safety to nonsense microdetails does our specialty a disservice. Residents constantly complain that the test material seems random and is not found in most review materials. This means either the Core Exam treats this material poorly or that the residents are studying the wrong information.

The problem is that this material is important. The ABR needs to make it clear what information candidates should know and release it as a packet of specific information like non-interpretive skills (NIS). In its current form, the combination of physics/radiation biology/radiation safety/nuclear medicine/RISE is a limitless almost black-box from which residents have no idea what to focus on or what material is high-yield. The end result is that most radiologists are taught low-yield or confusing information from physicists and end up with a poor understanding of these concepts. Candidates simply don’t really know what they should know and so don’t really know anything.

MR safety and contrast safety are included in the NIS study guide already (in addition to mission-critical information like the ACGME core competencies and how to create a “Culture of Safety”). The vast majority of the information I just described is also “non-interpretive” and needs to be included.

Judge Alonso dismissed the amended class-action lawsuit against the ABR on January 8, continuing the trend of denying a duel by trial in the ongoing saga of various doctors against the ABMS hegemony. You can read his opinion (~15 pages) here (and my most recent prior lawsuit-related post is here).

I can’t speak for its legal merits/basis, but as a non-lawyer, it’s pretty uninspiring. He largely turfs his interpretation to the prior ABIM lawsuit ruling.

The synopsis is that the judge is not convinced that initial certification (IC) and maintenance of certification (MOC) are different products because they were never sold separately. While IC was sold by itself, MOC was merely added decades later and never sold as a standalone product (never mind that a real monopoly would smartly generally avoid doing such a thing in real life).

This is probably a flaw in US antitrust law. When Microsoft got hammered for bundling Internet Explorer with Windows, Microsoft argued that IE was an inseparable part of Windows. This bundling crushed the browser war competition, further cemented Microsoft’s dominance, and stifled innovation for years. But part of what caused Microsoft to lose that argument was that IE was available on Mac, proving it could be a separate product. That detail, which in real life is functionally ancillary nonsense, was nonetheless key to proving Microsoft’s violation of the Sherman Antitrust Act.

Judge Alonso also doesn’t buy that there are separate markets for IC and MOC, not agreeing with the plaintiff’s new argument that MOC is essentially the same thing as other CME offerings. I agree that’s a hard sell, but he does discount that the NBPAS is selling its own MOC product as proof that separate demand for MOC exists (and that said NBPAS is struggling to compete given the ABMS monopoly in physician certification).

Would opening the gates to further scrutiny help? Should there be a chance to develop evidence? Would seeing some raw material from the ABR change the narrative?

It is true, as plaintiff argues, that many of the cases on which both parties rely throughout their briefs were decided, unlike this case, at the summary judgment stage or later, after the parties had the opportunity to develop evidence in discovery.

But then Alonso goes on to say he doesn’t believe the evidence will bear out the claims, so he’s not interested in seeing it. And even if it did:

On top of all of this, even if the Court assumes that initial certification and MOC are separate products, the Court still fails to see in what sense the tying arrangement alleged here poses a risk of foreclosure of competition in the tied market. “The traditional antitrust concern with [a tying arrangement] is that if the seller of the tying product is a monopolist, the tie-in will force anyone who wants the monopolized product to buy the tied product from him as well, and the result will be a second monopoly.”

The argument here is that essentially only MOC from the ABR has value because the ABR has a monopoly on IC. That MOC, even if untied, would be its own monopoly, because MOC from any other entity in this context would be meaningless. This is probably true in the current regulatory climate but is also guaranteed by rulings like Alonso’s.

Ultimately, the issue in medical certification isn’t necessarily the tying of MOC. It may be meaningless rent-seeking, but it may well be within the purview of the ABMS member boards. The underlying problem is that the complex regulatory and credentialing environment now includes (historically optional) certification to such an extent that no competitor can feasibly provide an alternative to initial certification. And this lack of competition and accountability has led to bloated organizations with outsized revenues and debatable value.

Although the Court doubts at this point that plaintiff will be able amend the complaint to state a claim, it cannot say so with certainty, so the dismissal is without prejudice and with leave to amend. Alternatively, plaintiff may elect to stand on the amended complaint and ask the Court to enter a final and appealable judgment.

And so, round two formally concludes.

Any amended complaint is due by February 5, 2021, but I suspect this may be the end given the legal costs. One guy against a monolithic organization with deep pockets was always going to be a tough battle. I doubt there’s much more here to amend for Alonso’s sake, but certainly appealing a final judgment to a higher court would be conceivable.

I’ve been talking to the ABR for a bit recently about OLA scoring. I’ve made no secret about my opinions on the actual content of the ABR’s 52-question-per-year MOC qbank and the likelihood of its ability to ensure anything meaningful about a radiologist’s ability to practice.

But I’ve been especially interested in how OLA is scored. For quite a while there was a promise that there would be a performance dashboard “soon” that would help candidates figure out where they stood prior to the big 200-question official “performance evaluation.”

And now it’s here. And, to their credit, the ABR released this blog post–How OLA Passing Standards and Cumulative Scores are Established–that actually really clarifies the process.

How Question Rating Works

In short, the passing threshold is entirely based on the ratings from diplomate test-takers, who can voluntarily rate a question by answering this: “Would a minimally competent radiologist with training in this area know the answer to this question?”

I think that wording is actually a little confusing. But, regardless, the fraction of people who answer yes determines the Angoff rating for that question. The passing threshold is above that percentage and failure is below, averaged out over all the questions you’ve taken. Because different people get different subsets of a large question database, currently “the range of passing standards is 50% to 83%.”

That’s a big range.

One interesting bit is that a question isn’t scorable until it’s been answered by at least 50 people and rated by 10, which means that performance can presumably change for some time even when not actively answering questions (i.e. if some already answered questions are not yet scorable when first taken). Back in the summer of this year, approximately 36% of participants had rated at least one question. It may take some time for rarer specialities like peds to have enough scorable questions to get a preliminary report.

The Performance Report

The current performance dashboard is a little confusing. Here’s mine:

It includes neither the actual passing threshold nor your actual performance. It’s got a cute little graph (with currently only one timepoint), but it reports performance simply as a percentage above or below the passing standard. I wasn’t quite sure what that meant mathematically, but the ABR actually addressed that in the post:

If the participant is meeting or exceeding that passing standard, their current cumulative score will indicate what percentage above the passing standard has been achieved. For example, if the average rating of the questions is 70 percent, the participant is required to answer at least 70 percent of the questions correctly within the set to meet the passing standard. If a participant’s current cumulative score indicates that they are 21 percent above the passing standard, then the participant would have successfully answered 85 percent of their questions correctly.

Based on that paragraph, their math here for passing individuals:

your percentage correct = passing threshold*percentage above + passing threshold = passing threshold*(1+percentage above)

In their example, that’s 85 = 70*.21 + 70 = 70*1.21

I’d rather just see the underlying numbers because the ABR doesn’t make it easy to see your cumulative performance at a glance. It’d be nice to see the actual passing threshold and your actual percent. But with this information, you can now–with some effort–estimate these numbers.

First, get your own raw performance. This isn’t straightforward. You’ll need to go to “My OLA History.” Filter to count your incorrect questions by clicking on “Your Answer” and selecting “incorrect.” Manually count that hopefully small number of questions and use that versus the total number you’ve answered (if you’ve been doing more than 52 per year, it’s going to be tedious to count manually).

So, for example, I’ve answered 104 questions and gotten 3 incorrect, which nets me a performance of 97%. My percentage above passing is reported as 50%, though it’s worth noting that the scale maxes out at plus/minus 50, which mathematically can’t be correct if the minimum passing threshold can be as low as 50%, so who knows. Anyway:

97 = X*1.5

X = 97/1.5 = 64.667

So we can estimate that I needed to get ~65% correct to meet the passing threshold given my question exposure, which is right in the middle of the current reported range.

Caveats include the fact that you don’t know how many if any of your questions are currently non-scoreable nor is it easy to see how many if any questions were subsequently nulled due to being deemed non-valid, so this really is just an estimate.

In related news, it’s lame there isn’t a way to see exactly how many questions you’ve answered and your raw performance at a glance. I assume they are trying to simplify the display to account for the varying difficulty of the questions, but it feels like obscuration.

Self-governance

The ABR has stated that the OLA question ratings from the community are all that’s used for scoring. While volunteers, trustees, etc are all enrolled in OLA and presumably rating questions, the ABR says what the greater community says is what goes.

So we are to judge what a radiologist should know based on what about 1/3 of MOC radiologists think a radiologist should know. There is no objective standard or aspirational goals. Radiologists can cherry-pick their highest-performing subspecialties and maintain their certification by answering a handful of questions in their favored subjects that are essentially vetted by their peers, who may not exactly be unbiased in the scoring process.

I don’t actually necessarily see a problem with that.

What I do see is more evidence of a bizarre double standard between what we expect from experienced board-certified radiologists and what we expect from trainees.

In court, the ABR has argued that initial certification and maintenance of certification are two aspects of the same thing and not two different products. As someone who has done both recently, they don’t feel the same.

Unlike OLA, neither residents nor the same broad radiology community has any say in what questions meet the minimal competence threshold for the Core and Certifying exams. For those exams, small committees decide what a radiologist must know.

You earn that initial certification and suddenly all of that mission-critical physics isn’t important. Crucial non-interpretive skills are presumed to last forever. And don’t forget that everyone should know how to interpret super niche exams like cardiac MRI (well, at least for two days).

It either matters or it doesn’t.

Why does physics, as tested by the ABR (aka not in real life), matter so much that it can single-handedly cause a Core exam failure for someone still in the cocoon of a training program but then suddenly not matter a lick when the buck stops with you?

The fact is that most practicing radiologists would not pass the Core Exam without spending a significant amount of time re-memorizing physics and nuclear medicine microdetails, and the new paradigm of longitudinal assessment doesn’t allow for once-per-decade cramming necessary to pass a broad recertification exam.

Meanwhile, diplomates are granted multiple question skips, which while designed to give MOC participants the ability to pass on questions that don’t mesh with their clinical practice can nonetheless be used to bypass questions that you don’t know the answer to. Especially for the performance-conscious and those with borderline scores, this creates a perverse incentive to miss out on rare learning opportunities.

But my point here is not that OLA should be harder.

My point is that certification should mean the same thing with the same standard no matter how long you’ve held it.