How the ABR’s MOC OLA scoring works

I’ve been talking to the ABR for a bit recently about OLA scoring. I’ve made no secret about my opinions on the actual content of the ABR’s 52-question-per-year MOC qbank and the likelihood of its ability to ensure anything meaningful about a radiologist’s ability to practice.

But I’ve been especially interested in how OLA is scored. For quite a while there was a promise that there would be a performance dashboard “soon” that would help candidates figure out where they stood prior to the big 200-question official “performance evaluation.”

And now it’s here. And, to their credit, the ABR released this blog post–How OLA Passing Standards and Cumulative Scores are Established–that actually really clarifies the process.

How Question Rating Works

In short, the passing threshold is entirely based on the ratings from diplomate test-takers, who can voluntarily rate a question by answering this: “Would a minimally competent radiologist with training in this area know the answer to this question?”

I think that wording is actually a little confusing. But, regardless, the fraction of people who answer yes determines the Angoff rating for that question. The passing threshold is above that percentage and failure is below, averaged out over all the questions you’ve taken. Because different people get different subsets of a large question database, currently “the range of passing standards is 50% to 83%.”

That’s a big range.

One interesting bit is that a question isn’t scorable until it’s been answered by at least 50 people and rated by 10, which means that performance can presumably change for some time even when not actively answering questions (i.e. if some already answered questions are not yet scorable when first taken). Back in the summer of this year, approximately 36% of participants had rated at least one question. It may take some time for rarer specialities like peds to have enough scorable questions to get a preliminary report.

The Performance Report

The current performance dashboard is a little confusing. Here’s mine:

It includes neither the actual passing threshold nor your actual performance. It’s got a cute little graph (with currently only one timepoint), but it reports performance simply as a percentage above or below the passing standard. I wasn’t quite sure what that meant mathematically, but the ABR actually addressed that in the post:

If the participant is meeting or exceeding that passing standard, their current cumulative score will indicate what percentage above the passing standard has been achieved. For example, if the average rating of the questions is 70 percent, the participant is required to answer at least 70 percent of the questions correctly within the set to meet the passing standard. If a participant’s current cumulative score indicates that they are 21 percent above the passing standard, then the participant would have successfully answered 85 percent of their questions correctly.

Based on that paragraph, their math here for passing individuals:

your percentage correct = passing threshold*percentage above + passing threshold = passing threshold*(1+percentage above)

In their example, that’s 85 = 70*.21 + 70 = 70*1.21

I’d rather just see the underlying numbers because the ABR doesn’t make it easy to see your cumulative performance at a glance. It’d be nice to see the actual passing threshold and your actual percent. But with this information, you can now–with some effort–estimate these numbers.

First, get your own raw performance. This isn’t straightforward. You’ll need to go to “My OLA History.” Filter to count your incorrect questions by clicking on “Your Answer” and selecting “incorrect.” Manually count that hopefully small number of questions and use that versus the total number you’ve answered (if you’ve been doing more than 52 per year, it’s going to be tedious to count manually).

So, for example, I’ve answered 104 questions and gotten 3 incorrect, which should me a performance of 97%. My percentage above passing is reported as 50%, though it’s worth noting that the scale maxes out at plus/minus 50, which mathematically can’t be correct if the minimum passing threshold can be as low as 50%, so who knows. Anyway:

97 = X*1.5

X = 97/1.5 = 64.667

So we can estimate that I needed to get ~65% correct to meet the passing threshold given my question exposure, which is right in the middle of the current reported range.

Caveats include the fact that you don’t know how many if any of your questions are currently non-scoreable nor is it easy to see how many if any questions were subsequently nulled due to being deemed non-valid, so this really is just an estimate.

In related news, it’s lame there isn’t a way to see exactly how many questions you’ve answered and your raw performance at a glance. I assume they are trying to simplify the display to account for the varying difficulty of the questions, but it feels like obscuration.


The ABR has stated that the OLA question ratings from the community are all that’s used for scoring. While volunteers, trustees, etc are all enrolled in OLA and presumably rating questions, the ABR says what the greater community says is what goes.

So we are to judge what a radiologist should know based on what about 1/3 of MOC radiologists think a radiologist should know. There is no objective standard or aspirational goals. Radiologists can cherry-pick their highest-performing subspecialties and maintain their certification by answering a handful of questions in their favored subjects that are essentially vetted by their peers, who may not exactly be unbiased in the scoring process.

I don’t actually necessarily see a problem with that.

What I do see is more evidence of a bizarre double standard between what we expect from experienced board-certified radiologists and what we expect from trainees.

In court, the ABR has argued that initial certification and maintenance of certification are two aspects of the same thing and not two different products. As someone who has done both recently, they don’t feel the same.

Unlike OLA, neither residents nor the same broad radiology community has any say in what questions meet the minimal competence threshold for the Core and Certifying exams. For those exams, small committees decide what a radiologist must know.

You earn that initial certification and suddenly all of that mission-critical physics isn’t important. Crucial non-interpretive skills are presumed to last forever. And don’t forget that everyone should know how to interpret super niche exams like cardiac MRI (well, at least for two days).

It either matters or it doesn’t.

Why does physics, as tested by the ABR (aka not in real life), matter so much that it can single-handedly cause a Core exam failure for someone still in the cocoon of a training program but then suddenly not matter a lick when the buck stops with you?

The fact is that most practicing radiologists would not pass the Core Exam without spending a significant amount of time re-memorizing physics and nuclear medicine microdetails, and the new paradigm of longitudinal assessment doesn’t allow for once-per-decade cramming necessary to pass a broad recertification exam.

Meanwhile, diplomates are granted multiple question skips, which while designed to give MOC participants the ability to pass on questions that don’t mesh with their clinical practice can nonetheless be used to bypass questions that you don’t know the answer to. Especially for the performance-conscious and those with borderline scores, this creates a perverse incentive to miss out on rare learning opportunities.

But my point here is not that OLA should be harder.

My point is that certification should mean the same thing with the same standard no matter how long you’ve held it.




ABR & Guessing the Cost of a Lawsuit

All non-profits have to file a Form 990 with the IRS detailing their finances. The ABR’s 990 says “THE BYLAWS, CONFLICT OF INTEREST POLICY AND FINANCIAL STATEMENTS ARE AVAILABLE UPON REQUEST.” I’ve already read and discussed the bylaws, but I thought I’d ask for the financial statements. Two emails went unanswered, but after I asked publically on Twitter I got a polite and professional email within a day.

Unfortunately, the statement I received was a broad profit and loss statement even less detailed than the 990. I’m not going to lie, I was really hoping they would send me something more granular that would further break down categories like travel to get a feel for how the ABR really operates. Travel expenses would likely include paying for coach airfare for volunteers to come together for question writing committees and the magical Angoff process, but they might also contain expenses related to annual getaways to Hawaii for the board. I don’t begrudge a working vacation, but big categories undeniably make it difficult to evaluate financial stewardship. Trolls on the internet talk a lot of smack about the ABR’s supposed largesse, but all we’ve really seen is a generous chief executive salary, a large pile of money in reserve, and some broad expense categories that I’d love to drill down. Large boxes hide their contents.

But since we can’t break down the big boxes, we may never know details the radiology community is interested in seeing. One recent example would be, how much is the ABR paying to fight off that class-action lawsuit?

The best we can do is a wild guess because all “legal” expenses are a single category in the ABR’s publically available tax documents (most recent filings are available on Guidestar).

Legal fees according to ABR Form 990:
2011: $57,280
2012: $70,811
2013: $78,271
2014: $114,563
2015: $44,776
2016: $48,703
2017: $45,439
2018: $25,294
2019: $119,445

We can see that earlier in the last decade, the fees were all over the place but mostly in the high five figures with exception of 2014. We then had several years in a row of lower numbers, primarily in the $40k range.

The initial complaint in the class action ABR lawsuit was filed on February 26, 2019, and the case is still ongoing.

The reported legal fees in 2019 were $119,445.

The average of the preceding 4 years prior to 2019 was $41,051.

If the costs of the lawsuit were responsible for the difference, that would be approximately $80k to fight the lawsuit in 2019 over 10 months. Is all that excess actually the lawsuit? Who knows; I don’t think they were sued in 2014 and that was a pricey year as well. Some likely additional one-time fees that I can think of like trying to deal with the legalese debacle of the ABR agreement earlier this year won’t appear until the 2020 Form 990 that will be filed next year. But we definitely have an upper bound.

The ABR’s legal counsel has filed three motions dated 6/27/2019 (54 pages), 03/13/20 (54 pages), and 07/21/20 (26 pages). It would seem likely that the overall cost will be at least double the 2019 amount if not substantially more. Just extrapolating on page count would put the estimate at $200k so far (though I would venture the research for the initial motion to dismiss would have taken longer and cost more).

While the case seems destined for dismissal, certainly an actual trial would increase costs exponentially. These lawyers presumably don’t charge for value like radiologists; they charge for time.

In 2019, there were (according to the ABR) approximately 31,200 diplomates paying for MOC (the very thing the lawsuit is about). Our very broad completely unscientific estimate would therefore suggest that each MOC-compliant radiologist, through their annual fee, paid about $2.50 in 2019 against their own interests (depending on whose side you take), which is less than 1% of their dues and which is, if we’re being honest, a trivial sum.

If the judge dismisses the current amended complaint and the case is subsequently dropped without further back and forth, then a non-grandfathered MOC-radiologist might expect to have contributed the equivalent of a beverage of indeterminate size and composition to support the ABR’s hegemony.

Considerations for Your First Job in Radiology

The majority of radiology trainees will take jobs in private practice, but whether academics or PP, there are still a lot of details that will significantly change what your job is actually like and what you should consider in choosing.

This post is very long (~3600 words), but I don’t feel like trying to publish it as a 10-part-series for more clicks. Refill your coffee or tea and let’s go.


Practice Setting

I’ve written previously about the differences between private practice and academic radiology, and I still largely agree with myself and that think post is worth reading. Go ahead, I’ll wait.

Private Equity

I don’t really want to get into this topic too much here. PE firms like to gobble up practices in a tight geographic area such that they can achieve local market dominance and then negotiate for higher reimbursement. Local monopolies are a business model. However, the easiest way for them to make money for their investors after a leveraged buy-out is by hiring more young people fresh out of residency for whom any attending salary seems high and using that relatively cheap labor to replace the older higher-paid partners that will soon be retiring on the cash (and potentially stock) from the buyout in a few years when everything vests.

If you need to live somewhere and that’s the shop in town, so be it. It’s not as though there aren’t bad groups that aren’t owned by private equity.

But on the whole, I think doctors do better for themselves and their patients when their personal and practice interests align. While you may have a limited voice as an individual radiologist working in a large group, an independent practice itself can choose its priorities and how those align for optimal patient care. A third party middle man dictating how much, how hard, and how fast you work over the long term is ultimately dictating how you practice medicine. I don’t think that’s good for our profession.

If you’re interviewing at an independent group, I would absolutely discuss the issue of a practice sale with them and see how they respond. Being in the work-up during a buyout is frequently a crappy situation to be in, where you’ve put in sweat equity only to have the benefit rug pulled out from underneath.

PE firms got a lot of bad rap in the news recently for things like surprise billing, and there’s an excellent reason for that. It’s because they deserve it.

Nighthawk & Teleradiology

Strict teleradiology is overall probably not an ideal first job after training. Being alone (and often production-based) fresh out of fellowship could be pretty lonely, disorienting, and stressful.

Night gigs are tough. Burnout is high and most people can’t maintain it for more than a few years. I’ve heard it can sometimes be hard to land a normal day job for some folks after a period of tele nights, and many groups treat their internal nighthawks as sort of second-class citizens. Typically the schedule and pay are good on paper, often 1 week on 1 week off (sometimes 1-on 2-off, which sounds much more sustainable), but it’s not uncommon for nighthawks to be strictly non-partnership-track employees with no avenue to transitioning to day work when you’re burnt. If you’re considering an internal nighthawk job, I would try to find a group that will work with you on these key details.

My group for example has a three-week rotation for our night radiologists where they work one week of days, one week of nights, and one week off. This way they’re part of the group and also do some regular work with other humans during normal human hours.

That said, there are people from whom this lifestyle is a perfect fit. If this is your plan you need to exhaust all of the options because not all remote work is created equal
(including both local group coverage and actual teleradiology companies). The case-mix and RVUs of an average shift vary widely. An 8-hour shift reading 60 RVUs is a completely different gig than 10-hour 100 RVU churns, even if you’re working 1-on-1-off.


Many residents/fellows will take a job locally, which likely reflects a combination of inertia and true preference. These sorts of opportunities are more straightforward to find and feel out. When groups are hiring, they typically reach out to local programs and will often be able to vet you through their personal contacts (even if those aren’t the people you’ve listed as your references). For your benefit, you’ll likely know (or can be put in contact with) former residents working at these groups that can give you the low down. Interviews won’t require travel. The whole thing will hopefully be pretty straightforward.

Finding nonlocal jobs may require a bit more work to suss out good opportunities. Some but not all positions will be posted on the ACR Career Center. Consider joining the American Radiologists Facebook group (and probably RadChicks for female rads), where openings also sometimes appear (and where you can query the hive [even anonymously] with job-related questions). I grabbed a body imager for our group via Facebook last year. This is the world we live in.

For better or worse, COVID means that a lot of groups are interviewing virtually. If you know what you want (say you really want to be a certain location and only one group is hiring), then this will at least be convenient, but overall I think it makes things harder in terms of assessing fit on both ends. Regardless of interview modality, you need to find some junior people in the group who have been there 0-2 years (or someone who just made partner if applicable) to talk with. Hopefully, that’ll be part of the interview process but these folks will provide you with an important perspective. Ask questions.


It is increasingly common in large cities with hyper-consolidated markets for IR to do basically all body IR and vascular IR procedures. Even in the common situation where others (such as a neuroradiologist) may perform inpatient lumbar punctures and myelograms during the day, it is not uncommon for any after-hours LPs to also go the way of the interventionalist. Smaller groups may still have general radiologists who perform a variety of nonvascular interventions like abscess drainages, and it’s still very common for groups to need the random rad on-site at various imaging centers to take care of the usual bevy of LPs, myelograms, arthrograms, and joint injections that may be scheduled there. But it’s also common for centers to schedule specific exam types on the days where the relevant subspecialists are covering or to only schedule certain types of procedures at specific centers. Most groups that need such coverage will ask you if you’re comfortable doing the things they need you to do. (If you’re not and you want to work there, the answer is that you’re happy to (re)learn).

Many academic groups also split procedures up by subspecialty such that a neuroradiology fellow would likely perform more spine biopsies and injections during training than they ever would once out in practice, where this may be handled by either interventional or neuro-interventional radiologists. Where I trained, for example, every division handled their own procedures, with the exception of lung biopsies, which switched from chest to IR during my first year in my residency. The bottom line is, it varies. And while this is subject to change, you should at least be aware of your comfort level and interests and the expectations of any group you look at.

For example, my fellowship was relatively procedure heavy, and I could easily be credentialed in the full gamut of spine interventions and pain injections, but in my actual practice, I do LPs, myelograms, GI/GU fluoroscopy, arthrograms, and joint injections. Meanwhile, our mammo is 100% performed by breast radiologists (whereas some places want everyone to do some breast imaging).

The broader your skill set, the more marketable you will be in general because different practices have different needs. But in real life, your practice may or may not be considerably more narrow. There’s also a decent chance that your first job won’t work out for one reason or another, so you could easily find yourself in a different situation doing some CME course and watching YouTube trying to brush up on skills you let atrophy. I once knew a 100% subspecialized academic MSK-radiologist who after like 7 years of practice transitioned to a nighthawk ER job to be able to spend more time with her kids. These things happen.

Salary & Vacation

Depending on any geographical constraints you may have, the details of group logistics may be moot. If you need to work in a certain city and there are three major groups of which only one is hiring in your subspecialty, then you will likely take a trap regardless of the size of the buy-in or if you think the call is equitable for people in the work-up.

Even more than salary, vacation is often the biggest measurable difference between academic practices and most private groups. Somewhere in the range of 8 to 12 weeks off is not uncommon in radiology private practice with less “desirable” locales sometimes offering even more (I’ve seen as high as 20 weeks at least several years ago, which is bonkers).

Large metros have seen a lot of consolidation over the past decade, and there are generally very few small groups left. The smaller groups where everyone practices as a complete generalist and does all the “light IR” are much more common in smaller cities and rural areas. Likewise, the consolidation and competition for imaging contracts have also pushed salaries down in big metros. You’re generally going to make more money with more vacation in a non-premier location within a region as well as in certain regions of the country more than others (e.g. midwest > coasts).

Your employee pay will definitely vary, but in many cases, it’s really the partner reimbursement where you’ll typically see the greatest spread. Whenever you evaluate money, make sure to include 401k profit-sharing contributions, group contributions to a health-savings account (HSA) for high-deductible health plans, and other such non-salary compensation. These add up and can make a huge difference.


In private practice, partnership is a big deal and you need to evaluate exactly what partnership means in your group, how long it takes to get there, and what buy-in you’ll need once you do. Some groups are completely democratic where all partners have a vote in major issues, all are eligible to serve on the board of directors (if there is one), and all get paid the same amount of money for the same amount of work, have the same amount of vacation etc. Others may have a tiered partnership similar to some law firms, where junior partners basically get a salary increase but senior partners still skim the cream financially and make the decisions. Still others are actually owned by private equity or another corporate entity and a “partnership” may or may not mean very much.

Some older radiologists may tell you that if a partner-track work-up is longer than a year then it’s BS and you should look elsewhere. That is unfortunately not the case in many markets, where you can easily expect tracks as long as 3-years. While the radiology job market is still much better than it was in the mid-teens, competition for hires is I think very unlikely to bring things back to what was common during the golden age of radiology reimbursement.

I also want to make it clear that an employed (non-shareholder track) position is not inherently bad. In some cases, an employed position can be more flexible and not have the same call responsibilities, can be part-time, or sometimes even get paid more (upfront) than a similar partner-track position in the work-up. In other groups, all non-partner employee jobs are the same whether you make partner after the prescribed amount of time or not. It’s absolutely possible to work for a good group as an employee just as it’s possible to be a partner in a crappy group.

If you’re in a group that offers both types of positions and decide to try to switch to partner-track, classically the work-up clock resets. Getting credit for time served is a point very much worth negotiating.


In most radiology practices, employees in the workup are already paying into the practice for the shareholders via sweat equity: you make less (and often have less vacation) than a partner for a period of time, and the group makes money off of you by paying you less than you bring in via production. Buying actual “shares” in the practice is usually on top of that, and the buy-in sum depends on the assets at stake. In a group without real estate/imaging centers, your buy-in is essentially for accounts receivable (the amount of money the group has earned/charged but hasn’t yet received). An AR buy-in will depend on the size of the group but should typically be in the five-figure range. A six-figure buy-in should start containing something more substantial.

The buy-in cost can often be financed by the group itself so that part of your new higher partner salary goes toward paying for the buy-in. In this case, your income bump will be attenuated for a while after you make partner.

If the buy-in is super high but the group’s assets suck, this is usually a way to get young partners to overpay for their slice of the pie, give the group more cash on hand, and then fund the exit-packages/retirement of the retiring partners when they sell their shares back to the group.

In general, a large democratic group is going to be more likely to have a fair valuation and should always share the numbers they use to derive everything. Theoretically, your buy-in should basically be the value of all the group assets (buildings, scanners, etc) divided by the number of partners (or the number of total shares multiplied by the number of shares you purchase when you become a partner). There shouldn’t be secrets. If the number makes you uneasy or you think the valuation of the assets looks too high, have someone evaluate the deal. You don’t want to be paying luxury car prices for a used Ford Pinto. And, at least to me, shady accounting is a red flag.

That said, your true “buy-in” isn’t just the lump sum you pay when you become a partner but is the combination of that and the difference in salary and vacation during the workup period. This can actually be hard to really define since partners are usually paid in a combination of salary, potentially “call pay” for working evenings and weekends, and profit-sharing. Suffice to say that employees are good for the group’s bottom line.

Going Part-time

One potentially important consideration that people often don’t discuss is how the group handles part-time work. Some groups allow partners to go down to 80% or sometimes as low as 50% time while still maintaining partnership status (you typically pay back your portion of the salary and benefits as a fraction of the full-time equivalent total). In these cases, it’s possible to continue making the same amount (or more) than you did as an employee while working considerably less.

Other groups don’t, which means that if you make the choice to roll back your hours in the future, you may need to do so as an employee of the group instead of a shareholder, typically a big hit to your reimbursement even on a pro-rated per-hour basis. In some cases you might also get less vacation than you did as a partner, negating some of the benefits of going part-time in the first place.

For reasons I don’t fully agree with, many groups are optimized for partner income and not set up to account for lifestyle flexibility. If you think you might want to work part-time at some point in the relatively near future, choosing a group that does not meaningfully allow for this may be a mistake.

Timing & Interviews

Try to interview at 3 or 4 places to be able to compare possibilities and make an informed decision. Know that it’s very common for a group to request an answer to a formal offer ridiculously quickly. At one group I was given a response deadline of 6 days! So know that you’ll need to cluster interviews if possible. You also just need to be honest with groups. If you have another interview the following week and you get an offer that’s asking for an immediate response, be gracious, thank them, and ask for more time. Don’t get muscled.

In recent years at least, lots of folks have started looking for jobs in the summer as fellowships started and had locked in jobs by early fall. While good positions may open up later, don’t wait to start looking.

As for the interview itself, you’ve done this a few times now. The same rules apply. In a large group, you may do most of your upfront communication and coordination with a practice administrator or even a dedicated practice recruiter. You know, be nice. If you’re interviewing somewhere academic, you’ll be giving a resident lecture, so prepare one that doesn’t suck.

Contracts & Negotiation

Offers will often come in three stages:

  1. Verbal
  2. Letter of Intent or Letter of Understanding
  3. Contract or Employment Agreement

Verbal agreements are supposed to count, but memory is imperfect. Get everything important to you written down.

The Letter of Intent is a short binding contract, often 1-2 pages, that in many cases is the only document that actually specifies the core details of your job, such as your salary, vacation, section/division, and often (current but subject to change) call responsibilities. Make sure everything important is in there before you sign it. If you want something spelled out in writing, in many cases it will actually make more sense for it to show up in a Letter of Understanding than it would in the formal Employment Agreement (which will often reference the Letter).

In many radiology practices, the Employment Agreement is standard across the group and will be the one document of the three that’s written in legalese. It typically details boring important things like how the group handles non-competes (aka “restrictive covenants”), grounds for termination, intellectual property, malpractice insurance, etc. You need to understand this material, and if you don’t, you need to hire someone who can translate.

As a practical matter, there often isn’t a lot of wiggle room on the main numbers for a large radiology group as compared with what your clinician colleagues often deal with. Salary and vacation are typically standard. Buy-in will also be set at the time of signing. Even most of the paperwork details like non-competes are often going to be universal to the group and therefore completely non-negotiable. If every partner has agreed to something, you’re probably not going to get out of it. A one-time signing bonus, however, will often have some wiggle room, especially if you’re coming from somewhere far away and could use extra help with moving expenses.

If you have intellectual property or plan to develop any, please be careful how that topic is described in your employment agreement. Academic centers, for example, particularly love to include heavy-handed language that they own everything and anything you come up with while employed, even when you do so outside of work hours. Likewise, while you may or may not be able to change a non-compete (and the group may or may not actually try to enforce it), the exact language will define how limiting it can be. Is a 5-mile radius defined from the group headquarters, from the main hospital(s) you read for, or from any imaging center you contract with? Because the latter will typically end up excluding the whole city.

I would argue the most important thing that you can negotiate is the makeup of your clinical duties. What fraction will you practice in your subspecialty? What procedures will you perform? Which hospitals and imaging centers will you cover? Will you read breast? If there is more than one divisional call-pool, which one(s) will you be in? Do you have to work nights? What’s the evening/swing shift and weekend burden? What fraction if any can be done from home? If there’s internal moonlighting, what types will be available to you?

To be clear, some or all of these types of details may be part of the job offering itself or equitable across the group, but I would venture that most people fixate on measurable things like salary size when comparing jobs and not nearly enough time evaluating the actual job.

There is no job in radiology that is so poorly paid that you can’t make a good living, but there are jobs so miserable that you can’t enjoy your life.

Again, while many details may not be meaningfully negotiable, these sorts of clinical parameters are worth discussing and feeling out. Don’t assume. Likewise, if you get promised something, get it in writing. You don’t want to be told you don’t have to cover some remote outlying hospital only to get assigned there the moment you show up. Your call responsibilities are likely subject to change, for example, but one thing to consider asking about is some type of credit if your call frequency is defined in your contract but subsequently increases beyond the original threshold. Wouldn’t it be nice if working an extra day at least moved up your partnership date?


Good luck.

Ask questions.

I’ve had friends whose groups have been bought out by PE firms, and I’ve had friends who have then rapidly left those jobs and others who have stuck it out. I’ve had friends who have been blindsided and not offered partnership in their groups, and I’ve had friends who didn’t wait long enough to find out because they didn’t like how radiology was being practiced.

If you find yourself in a position you truly don’t like, the prospect of more money in the future is unlikely to make that job suddenly tolerable. Your most valuable asset is your time, so don’t waste it doing something you hate.



Updates in the ABR Lawsuit

The back and forth continues. You can see the duel laid out on the lawsuit’s website, but they’re now up to 9 filings, most recently on August 12, when the plaintiff added their “Sur-Reply in Opposition to Motion to Dismiss” (which is an additional response to specific arguments made by the American Board of Radiology in their second motion to dismiss which itself was a response to the plaintiff’s amended complaint after the initial complaint was dismissed).

For those just tuning in, the plaintiff’s main thrust has been to argue that the ABR has abused an illegal monopoly by using their complete market dominance in essentially mandatory (initial) board certification to then force all radiologists to continue paying for the “separate product” of maintenance of certification forever. The ABR has argued that MOC is just part of certification now and is not a separate product, even if it wasn’t before and hasn’t been for the majority of the ABR’s history since its founding almost a century ago.

You don’t have to read much to see why the legal system is terrible, lawsuits take forever, and lawyers make a lot of money. The initial lawsuit was filed in February 2019. The amended complaint was filed in January 2020. And this is all quite possibly going to be thrown out by the judge again without even a hint of a trial after at least two years of the ABR burning through our certification fees to preserve its (new) status quo.

One excerpt from the end of this recent back and forth.


If Plaintiff’s conclusion were true, hospitals and other medical organizations would not require certification (which includes MOC) to the degree alleged by Plaintiff. Competing entities would be permitted to offer substandard CPD products at bargain prices, and ABR would be powerless to control the integrity of its certification.

Of course, most hospital credentialing specifically does not include MOC, and such a requirement is even illegal in some states. It’s only “required” in the sense that the ABMS boards are now saying they’ll revoke certification from those who have earned it if they don’t keep paying their annual tithe.


There is nothing [in the motion] about “substandard CPD products,” “bargain prices,” or “the integrity of [ABR’s] certifications.” While ABR may assert its illegal tying is a justified attempt to preserve the undefined “integrity of its certifications,” that inherently fact-driven affirmative defense relies on facts outside the pleadings, is inappropriate on a motion to dismiss, and is contradicted by Plaintiffs’ well-pleaded allegations that MOC does not benefit physicians, patients, or the public, or improve patient outcomes. For example, ABR does not contend ABR certified radiologists failed to satisfy the “integrity” of certifications before MOC; that the “integrity” of certifications was in decline before MOC; that making MOC mandatory protects the “integrity” of certifications; or that hospitals, patients, and insurance companies had less “trust” in certifications before MOC was made mandatory.

Plaintiff seeks only to break up the captive market ABR has created for MOC by tying it to certifications. Eliminating the illegal tie and making MOC voluntary as promised previously by ABR will allow the marketplace to decide the merits of MOC, as the antitrust laws require.

Presumably, the judge will weigh in again in the coming months. His initial response was to grant the ABR’s initial motion to dismiss.

Recalls and Exam Security

Out of all the reasons organizations like (but not limited to) the ABR have used as an excuse to shy away from remote content or historically relied on commercial testing centers, I strongly suspect exam security is the only one that actually matters.

While an individual not cheating on the exam is important for exam integrity, that type of exam security is a relatively straightforward n=1 problem. The real exam security that matters is the security of intellectual property. Nevermind that all these medical organizations use questions largely written and initially vetted by volunteers, losing hundreds of proprietary questions in one fell swoop to some industrious malcontent is the real fear.

The recent utter failure of the American Board of Surgery’s virtual testing also makes the point: once people have seen a significant fraction of the questions for a high stakes exam, it’s back to the drawing board. A doomed effort to administer an exam remotely sets an organization back by months, at the least.

That’s because recalls—people sharing memorized exam content—are a big deal. In fact, news about the universal use of recalled study questions for the radiology oral boards back in 2012 was the driving force behind the creation of the MCQ-only Core and Certifying exams, administered only on-site in bespoke testing centers created by the ABR itself in Chicago and Tuscon that sit gathering dust for most of the year.

While recalls are against organizational policy (and thus certainly something an individual should not do), the focus on recalls as a destabilizing force for the fairness of exams is…lame. Literally every important high-stakes exam including the SAT, MCAT, USMLE, and the various board exams have engendered a massive test prep industry around offering nearly the same thing: questions written–sometimes by the same people!–to exactly simulate those very same exams. Many, including most of the USMLE prep products, use software that even completely mimics the test software down to the pixel.

But I want to posit a fundamental misunderstanding:

A Really Meaningful Modern Test Shouldn’t Rely on Hiding its Content

Imagine a radiology exam had a question demonstrating a benign hepatic hemangioma on CT or MRI. Imagine a second-order question asking about management (nothing). If that information were put into a series of recalls, it would be meaningless. Every radiology resident should get the question correct because it is relevant to radiology practice and unavoidable during normal training. And if the exam, like the USMLE licensing exams or the various specialty board exams, purports to be a measure of minimal competency for safe independent practice, then all the questions should be relevant to daily practice.

If someone has mastered all of the relevant “recalls,” then they would presumably be ready to practice radiology. It’s okay if the fraction of answered questions is high if we expect the questions to reflect things we really want everyone to know. Mastering all of this exam content is exactly what we want trainees to do!

Recalls only matter in two situations:

  1. If exams need to include questions designed to differentiate high-level performance, separating the best from the average. In this setting, questions that should be really challenging become easy, throwing off the balance of exam difficulty. It’s precisely because the designers want to give you brainbusters that the element of surprise is key. But in a world where high-quality informational content is at your fingertips, this component of mastery is increasingly irrelevant. The things we really want people to know quickly, like how to manage a true emergency like a code or how to drop a line, are never satisfactorily going to be assessed with a multiple-choice question. That’s what a real-life training program is for. Performance on an MCQ test is typically only good at predicting future performance on other MCQ tests.
  2. Questions that really really test critical thinking. In this setting, the examinee shortcuts the thought process and arrives directly at the answer, defeating the purpose of the question. While the MCAT contains a fair number of reasoning questions along these lines, this is rarely the case in real life for medical licensing and board exams.

Every question bank for every major test is composed of glorified recalls. Pretending otherwise is silly. If all questions contained important material and the ability to answer them reflected meaningful knowledge and competence, then someone able to memorize the plethora of recalled questions would be exactly what you’d want: qualified.