Probably the biggest news in radiology over the past year (at least for residents) was the announcement that the upcoming and all future ABR examinations were moving to an online remote/virtual format. That’s worked out pretty well so far.

One bit of nice unexpected news that was announced very quietly this week was that the ABR Core Exam, the first and only meaningful component of the radiology exam certification cycle, would no longer have a separately-graded physics section that could—by itself—prevent overall exam passage. Physics will still comprise an unchanged amount of the test but will be graded as just another section along with all the rest: a component for overall passage but not a section that examinees can “condition” and be forced to retake at a later time.

Holding physics somewhat apart was a holdover from the pre-Core Exam era when there was a completely separated dedicated physics exam.

The ABR made this decision during the grading process just last week. I’m sure that recent examinees would have really appreciated this information during their studies, but timing aside I fully support the ABR’s choice here. Strong move. What’s next?

It’s residency Rank Order List season, and I thought I’d share a paper published in AJR from 2016 titled, “Radiology Resident’ Satisfaction With Their Training and Education in the United States: Effect of Program Directors, Teaching Faculty, and Other Factors on Program Success.”

It was a small study where the authors surveyed 217 radiology residents.

Of that group, 168/216 (77.8%) of residents were satisfied overall with their residency programs.

You’re always going to have some sour grapes, and it’s not possible from the data to figure out how much of that fraction might be related to noncontrollable factors like the city, the stresses of dealing with student loans, or other factors. But as the authors noted, that’s a big difference from the golden era:

This is lower than reported in similar previous national surveys conducted by the American College of Radiology, which reported a 97.8% level of job satisfaction of radiology trainees in 2003, and 97.6% level of job satisfaction in 1995.

Like with internet comments and product reviews, convenience surveys always lean toward the dissatisfied. But the data are still interesting because they can highlight the causes for dissatisfaction, even if they play an outsize role compared to the general community.

The three sub-categories with the greatest correlation with overall satisfaction were satisfaction with the program director and administrative office, daily workstation experience, and faculty.

So basically everything.

But of the three, the program director and administration were by far the most dominant. I suspect it’s more of a break not make scenario:

The factor with the greatest correlation to satisfaction with the program director and administrative office is how approachable and responsive the administration is to resident concerns.

A good program director may help make a program, but a bad one can definitely tank one.

A good PD is both a boss and an advocate. While not all trainee complaints are necessarily fixable (or even reasonable), accountability, transparency, and attentiveness aren’t something to take for granted in program leadership. Culture colors everything.

It might be impossible to change the culture of the hospital. It might be impossible to reduce RVU pressures on faculty or improve mediocre teachers. So the survey is actually good news because the program leadership is far more mutable.

There was one throwaway statistic they reference from a 2003 study that ties into the daily workstation experience/faculty components:

A survey with 132 junior radiologists revealed that 68% of the responders left academia after an average of 3.28 years because of low pay and lack of academic time.

That was almost 20 years ago, and the RVU pressure and lack of academic time have gotten worse since then. I wonder what that number is now.

My group is extremely stable, but I’ve seen a ton of turnover amounts young attendings in both academia and private practice. I don’t think enough practices of any variety are willing to allow for a Goldilocks approach between productivity and revenue.

The February 2021 issue of the BEAM features a short article with the title, “Board, Staff Working Together to Control Expenses.”

As the Board of Governors discussed these new [remote] exam tools, one of the perceived potential benefits was the intuitive opportunity to decrease costs and, by extension, reduce fees. However, there are persistent barriers to fee abatement at the time of this writing, including the absence of proven success of the new exam structure; a lack of dependable forecasts of the future steady-state expense structure; the inherent long-term nature of established financial obligations related to exam center equipment and leases; and the unexpected short-term development costs of the virtual exam platform software.

Proven success? Check.

Does any stakeholder believe that ~$50 million in cash reserves isn’t enough to deal with “a lack of dependable forecasts of the future steady-state expense structure” [sic]?

They continue (emphasis mine):

ABR senior leadership is committed to working with the board to control costs. We are optimistic that this is achievable as we close in on the “new normal,” but we don’t know the extent of potential cost reductions, nor when they might be achieved. The less visible infrastructure elements of board functions, ranging from cybersecurity to volunteer support, are critical to customer service for our candidates and diplomates, as well as fulfillment of our core mission. Despite these obstacles, the board members view themselves as responsible stewards of ABR resources, both financial and otherwise. In this vein, they consistently challenge each other, and the ABR staff, to reduce costs and, subsequently, fees, to the extent possible.

Transparency, transparency, transparency. Anything less is just self-love.

Last year as the pandemic spiraled out of control, the ABR resisted—as they have for years—calls for disseminated exams away from their centers in Chicago and Tuscon. The lack of a foreseeable endpoint and pressure from advocates was finally enough for the ABR to make the switch. And to their credit, when the ABR came around, they went all the way: all exams are to be virtual from this point forward.

And guess what? It worked.

Apparently, it worked really well.

And I, for one, am not surprised.

People I’ve spoken to were overall very pleased with the remote experience. Were there rare technical difficulties? Sure. But reports are that the ABR was generally responsive and helpful in aiding candidates when issues cropped up, and multiple residents I spoke to gave ABR customer service high marks.

So while perhaps they shouldn’t have needed the worst pandemic in a century to make these changes, credit where it’s due: the ABR successfully pulled off the transition to at-home testing.

The ABR’s testing centers, though physically inconvenient, were always pretty nice compared with most commercial centers. But the ability to take the exam from a location of your choosing with no travel required and your choice of preferred snacks, clothing, thermostat settings, and bathroom is pretty nice. Having the exam over three days also probably helped with test-fatigue.

Future Fix Requests

There were a few complaints the ABR should address in future administrations:

• Answer choice strikethrough. This was a common request, and it’s a common feature including one available on the USMLE exams that residents are all used to.
• Cine clip optimization. This has been a longstanding complaint, but in this case, at least sometimes clips are presented in a separate window from the question and answer choices. They should be embedded the same way as normal images with easily controllable playback speed and the ability to manually scroll.
• Remove the 30 question auto-lock. The need to lock previously seen questions makes perfect sense at the end of a 60-question block and whenever a candidate takes an optional break. But I’m not sure I buy any justification for auto-locking mid-section. This is a true functional change from prior exam administrations that has a negative impact on those who would like to review all related questions before moving on. It’s also difficult to know how much time to allot to question review when you break up 120 questions into 4 blocks instead of two, making time management more difficult.
• Announce the section order. This was a big complaint I heard and one I agree completely with. For years the ABR has avoided publicizing the section order (e.g. Breast, then Cardiac, then GI) despite keeping it consistent across testing administrations. While people obviously aren’t supposed to discuss the exam, in the real world this has meant that candidates taking the exam later always know what sections are coming on which days, allowing them to cram most effectively. Unless every candidate has a randomized order, keeping this information semi-hidden in this setting just isn’t appropriate and should be a no-brainer to fix. Knowing you’re going to have ridiculous radioisotope safety microdetails on a specific day means you can prepare for that much more effectively and seriously jeopardizes the exam integrity. Again, this is not a new issue.

The Core Exam is still the Core Exam

Ultimately, the biggest complaint—no surprise—wasn’t the software but the test itself. It’s not as though the content magically became more on-point just because you got to wear pajamas.

If I were to limit myself to one content suggestion, it’s this:

I feel very strongly that the ABR’s reduction of physics and radiation safety to nonsense microdetails does our specialty a disservice. Residents constantly complain that the test material seems random and is not found in most review materials. This means either the Core Exam treats this material poorly or that the residents are studying the wrong information.

The problem is that this material is important. The ABR needs to make it clear what information candidates should know and release it as a packet of specific information like non-interpretive skills (NIS). In its current form, the combination of physics/radiation biology/radiation safety/nuclear medicine/RISE is a limitless almost black-box from which residents have no idea what to focus on or what material is high-yield. The end result is that most radiologists are taught low-yield or confusing information from physicists and end up with a poor understanding of these concepts. Candidates simply don’t really know what they should know and so don’t really know anything.

MR safety and contrast safety are included in the NIS study guide already (in addition to mission-critical information like the ACGME core competencies and how to create a “Culture of Safety”). The vast majority of the information I just described is also “non-interpretive” and needs to be included.

Judge Alonso dismissed the amended class-action lawsuit against the ABR on January 8, continuing the trend of denying a duel by trial in the ongoing saga of various doctors against the ABMS hegemony. You can read his opinion (~15 pages) here (and my most recent prior lawsuit-related post is here).

I can’t speak for its legal merits/basis, but as a non-lawyer, it’s pretty uninspiring. He largely turfs his interpretation to the prior ABIM lawsuit ruling.

The synopsis is that the judge is not convinced that initial certification (IC) and maintenance of certification (MOC) are different products because they were never sold separately. While IC was sold by itself, MOC was merely added decades later and never sold as a standalone product (never mind that a real monopoly would smartly generally avoid doing such a thing in real life).

This is probably a flaw in US antitrust law. When Microsoft got hammered for bundling Internet Explorer with Windows, Microsoft argued that IE was an inseparable part of Windows. This bundling crushed the browser war competition, further cemented Microsoft’s dominance, and stifled innovation for years. But part of what caused Microsoft to lose that argument was that IE was available on Mac, proving it could be a separate product. That detail, which in real life is functionally ancillary nonsense, was nonetheless key to proving Microsoft’s violation of the Sherman Antitrust Act.

Judge Alonso also doesn’t buy that there are separate markets for IC and MOC, not agreeing with the plaintiff’s new argument that MOC is essentially the same thing as other CME offerings. I agree that’s a hard sell, but he does discount that the NBPAS is selling its own MOC product as proof that separate demand for MOC exists (and that said NBPAS is struggling to compete given the ABMS monopoly in physician certification).

Would opening the gates to further scrutiny help? Should there be a chance to develop evidence? Would seeing some raw material from the ABR change the narrative?

It is true, as plaintiff argues, that many of the cases on which both parties rely throughout their briefs were decided, unlike this case, at the summary judgment stage or later, after the parties had the opportunity to develop evidence in discovery.

But then Alonso goes on to say he doesn’t believe the evidence will bear out the claims, so he’s not interested in seeing it. And even if it did:

On top of all of this, even if the Court assumes that initial certification and MOC are separate products, the Court still fails to see in what sense the tying arrangement alleged here poses a risk of foreclosure of competition in the tied market. “The traditional antitrust concern with [a tying arrangement] is that if the seller of the tying product is a monopolist, the tie-in will force anyone who wants the monopolized product to buy the tied product from him as well, and the result will be a second monopoly.”

The argument here is that essentially only MOC from the ABR has value because the ABR has a monopoly on IC. That MOC, even if untied, would be its own monopoly, because MOC from any other entity in this context would be meaningless. This is probably true in the current regulatory climate but is also guaranteed by rulings like Alonso’s.

Ultimately, the issue in medical certification isn’t necessarily the tying of MOC. It may be meaningless rent-seeking, but it may well be within the purview of the ABMS member boards. The underlying problem is that the complex regulatory and credentialing environment now includes (historically optional) certification to such an extent that no competitor can feasibly provide an alternative to initial certification. And this lack of competition and accountability has led to bloated organizations with outsized revenues and debatable value.

Although the Court doubts at this point that plaintiff will be able amend the complaint to state a claim, it cannot say so with certainty, so the dismissal is without prejudice and with leave to amend. Alternatively, plaintiff may elect to stand on the amended complaint and ask the Court to enter a final and appealable judgment.

And so, round two formally concludes.

Any amended complaint is due by February 5, 2021, but I suspect this may be the end given the legal costs. One guy against a monolithic organization with deep pockets was always going to be a tough battle. I doubt there’s much more here to amend for Alonso’s sake, but certainly appealing a final judgment to a higher court would be conceivable.

With permission, I’m reposting a (very lightly edited) anonymous social media post from a young radiologist who joined a private practice that had recently been purchased by private equity:

I think I committed a huge mistake in signing up for a job with a large private practice group that was bought by a big private equity group in the radiology space. There has been a massive turnover of non-partner radiologists, over 30+% pay-cut in collections, very close monitoring of productivity, poor leadership, and no concern for younger radiologists. Almost everything told to me at the time of my interview turned out to be incorrect including work volume, projected compensation, and the reasons non-partners had left.

Several older partners are retiring as they made their millions in the buyout. The practice can’t hire fast enough as younger rads keep quitting or getting fired, so we’re overall chronically understaffed. I work extremely high volume (25k or more RVUs/year) extremely busy call with very low salary of $300k.

I fear there could be another sale of the practice in the future given how rapidly this private equity company is trying to acquire other practices, further driving down salary. We’d stay understaffed (many non partners leave after the buy-out), so volume will likely still be too high especially for the salary given. Could go on, but feeling really stuck. Any recommendations if I should stick it out or quit?

This is the private equity bait and switch, and I don’t even mean just in the premeditated awfulness of an operational model largely predicated on buying a business with the intent of squeezing the value-creating units (human beings) for more value by a combination of more work for less pay.

I mean that, in many cases, the natural history of these practices can result in a nonviable work environment through what should be an expected evolution of staffing changes.

Let’s walk through one way this can play out:

• Partners get a buy-out that doesn’t fully vest for a pre-specified duration of time.
• Some non-partners immediately leave the practice due to perceived or real insult, insufficiently generous retention package, or knowledge that long-term income will fall. Non-partners expect to pay sweat equity to become a shareholder, but they don’t want to work hard for less pay if there isn’t a meaningful long-term partnership at the end of it.
• Non-partner employee exodus results in immediate understaffing. With the same work and fewer people, everyone is taking more call and needs to work harder to keep up with the lists. Even a desirable practice can’t necessarily hire people instantaneously.
• The job is therefore less desirable and may have a hard time recruiting and retaining talented employees.
• The partners finally get all of their money. Many may have planned to retire anyway at this juncture but many more will certainly move on if the job now sucks, which it often does.
• Even more understaffing occurs.

Most PE practices haven’t gotten past this timeline yet. Potential outcomes are re-selling to a larger fund, selling the practice back to the original physician shareholders, and/or significant operational changes. In a chronic understaffing situation, employee pay sometimes becomes more competitive in order to retain FTEs. In some cases, the now underperforming practice may lose imaging contracts, which has the unanticipated benefit of fixing the understaffing problem.

Now to be clear, these issues can arise in any practice. There are certainly examples of physician-owned groups squeezing employees in the workup with the promise of partnership only to churn them when the time comes. Corporations and PE groups absolutely do not have a monopoly on being jerks. However, a once-profitable business is automatically less “profitable” when a third party inserts itself to take a mandatory cut. The value-add of oft-touted “productivity” gains can only take you so far, particularly in the face of downward reimbursement pressure.

And to be extra clear, someone just making you work harder and read more cases per day for the same pay isn’t the kind of efficiency gain any group should be proud of. While $300k is obviously a lot of money, the average radiologist reads in the ballpark of 10k RVU per year depending on subspeciality. 25k RVUs is a massive number, which means that the anonymous poster is generating a ton of money for someone else on the back of their misery.

Their job is almost certainly not going to get better, and they should leave.

Lesson: Know the group dynamics and local market of any practice you consider joining.

The report from the ACR Task Force on Certification in Radiology is out. This is the American College of Radiology’s formal take on how the American Board of Radiology is doing exercising its monopoly on radiology certification.

It’s clear, concise, well-researched, and contains wonderfully diplomatic language. I admire the restraint (emphasis mine):

Fees have remained unchanged for initial certification since 2016 and MOC since 2015. We acknowledge there is a cost of doing business and reserves are necessary but increased transparency and cost effectiveness are encouraged.

This in reference to finances like this. Such a gentle request.

Radiologists are also concerned that there is absence of scientific evidence of value.

An understatement certainly written like it was dictated by a radiologist.

We congratulate the ABR for modernizing its testing platform for MOC Part 3. The move to OLA is a responsive change from feedback. However, we are not aware of any theory or research that supports how the annual completion of 52 online multiple-choice questions (MCQ) demonstrates professional competence.

Ooph. Boom goes the dynamite.

MOC critique is tough.

On the one hand, OLA is better than a 10-year exam based on sheer convenience alone. It’s a trivial task, and therefore I know many radiologists don’t want to complain because they’re concerned that any changes would only make MOC more arduous or challenging (a valid concern). Organizations would much rather increase the failure rate to stave off criticism about a useless exam than actually get rid of a profit-generating useless exam (see USMLE Step 2 CS).

On the other hand, what a joke. There is literally no basis for assuming this token MCQ knowledge assessment reflects or predicts anything meaningful about someone’s ability to practice. Even just the face validity approaches zero. (Of course, this argument could also apply to taking 600+ multiple-choice questions all at once for initial certification).

Scores on standardized tests have been shown to correlate better with each other than with professional performance. Practical radiology is difficult to assess by MCQs, requiring a much greater skillset of inquiry and judgment.

This relates to the only consistent board certification research finding: standardized testing scores like Step 1 are the best predictors of Core Exam passage. People who do well on tests do well on tests. And while certainly smart hard-working people are likely to remain smart hard-working people, it remains to be seen if Step 1, in-service, or even Core Exam performance predicts actually being excellent at radiology versus being excellent at a multiple-choice radiology exam.

The obvious concern is that it’s the latter: that the ABR’s tests do not differentiate between competent and incompetent radiologists, and that we are largely selecting medical students based on their ability to play a really boring game as opposed to their ability to grow into radiologists.

Successful certification programs undertake early and independent research of assessment tools, prior to implementation. This is a vital step to ensure the accurate assessment of both learner competence and patient outcomes.

Subtle dig with the use of the word successful, but this is the crux:

Assessments are not bad. Good assessments are a critical component of the learning process. Bad assessments are bad because they provide incomplete, confusing, or misleading information, a problem compounded when a preoccupation with doing well on said bad assessment then distracts learners from more meaningful activities (look no further than Step 1 generated boards fever).

Medicine and radiology should not be limited by legacy methodology. Recognizing that learning and assessment are inseparable, the ABR has the opportunity to lead other radiology organizations, integrating emerging techniques such as peer-learning and simulation into residency programs. Assessment techniques are most effective when they create authentic simulations of learners’ actual jobs, although such techniques can be time-consuming and resource-intensive to develop.

Yes.

And I’ll say it again: diagnostic radiology is uniquely suited–within all of medicine–to incorporate simulation. Whether a case was performed in real-life years ago or is fresh in the ER, a learner can approach it the same way.

Despite alternative certification boards, the market dominance of the ABMS and its member boards has been supported by a large infrastructure of organizations that influence radiologists’ practices. The ABR should welcome new entrants, perhaps by sponsoring products developed by other organizations to catalyze evolution, innovation and improvement to benefit patients.

Hard to imagine that alternate reality.

Although the ABR meets regularly with leadership from external organizations, such as APDR, the ABR could better connect with its candidates and diplomates by reserving some voting member positions on their boards for various constituencies.

As I discussed in my breakdown of the ABR Bylaws, there is a massive echo chamber effect due to the ABR’s promotion policy, which requires all voting board members to be voted in by the current leadership, usually from within the ranks of its hard-working uncompensated volunteers. This means that operationally, the ABR is completely led, at all levels of its organization, by people who believe in and support the status quo.

Meeting with stakeholders may act as a thermometer helping them feel the room. The recent inclusion of Advisory Committees that give intermittent feedback and the perusal of social media commentary may provide the occasional idea. But all of this information is, by the ABR’s design, put into a well-worn framework.

The ABR is designed to resist change.

No one has a vote who wasn’t voted to have a vote by those who already vote.

And that’s a problem.

I’ve been talking to the ABR for a bit recently about OLA scoring. I’ve made no secret about my opinions on the actual content of the ABR’s 52-question-per-year MOC qbank and the likelihood of its ability to ensure anything meaningful about a radiologist’s ability to practice.

But I’ve been especially interested in how OLA is scored. For quite a while there was a promise that there would be a performance dashboard “soon” that would help candidates figure out where they stood prior to the big 200-question official “performance evaluation.”

And now it’s here. And, to their credit, the ABR released this blog post—How OLA Passing Standards and Cumulative Scores are Established—that actually really clarifies the process.

How Question Rating Works

In short, the passing threshold is entirely based on the ratings from diplomate test-takers, who can voluntarily rate a question by answering this: “Would a minimally competent radiologist with training in this area know the answer to this question?”

I think that wording is actually a little confusing. But, regardless, the fraction of people who answer yes determines the Angoff rating for that question. The passing threshold is above that percentage and failure is below, averaged out over all the questions you’ve taken. Because different people get different subsets of a large question database, currently “the range of passing standards is 50% to 83%.”

That’s a big range.

One interesting bit is that a question isn’t scorable until it’s been answered by at least 50 people and rated by 10, which means that performance can presumably change for some time even when not actively answering questions (i.e. if some already answered questions are not yet scorable when first taken). Back in the summer of this year, approximately 36% of participants had rated at least one question. It may take some time for rarer specialities like peds to have enough scorable questions to get a preliminary report.

The Performance Report

The current performance dashboard is a little confusing. Here’s mine:

It includes neither the actual passing threshold nor your actual performance. It’s got a cute little graph (with currently only one timepoint), but it reports performance simply as a percentage above or below the passing standard. I wasn’t quite sure what that meant mathematically, but the ABR actually addressed that in the post:

If the participant is meeting or exceeding that passing standard, their current cumulative score will indicate what percentage above the passing standard has been achieved. For example, if the average rating of the questions is 70 percent, the participant is required to answer at least 70 percent of the questions correctly within the set to meet the passing standard. If a participant’s current cumulative score indicates that they are 21 percent above the passing standard, then the participant would have successfully answered 85 percent of their questions correctly.

Based on that paragraph, their math here for passing individuals:

your percentage correct = passing threshold*percentage above + passing threshold = passing threshold*(1+percentage above)

In their example, that’s 85 = 70*.21 + 70 = 70*1.21

I’d rather just see the underlying numbers because the ABR doesn’t make it easy to see your cumulative performance at a glance. It’d be nice to see the actual passing threshold and your actual percent. But with this information, you can now—with some effort—estimate these numbers.

First, get your own raw performance. This isn’t straightforward. You’ll need to go to “My OLA History.” Filter to count your incorrect questions by clicking on “Your Answer” and selecting “incorrect.” Manually count that hopefully small number of questions and use that versus the total number you’ve answered (if you’ve been doing more than 52 per year, it’s going to be tedious to count manually).

So, for example, I’ve answered 104 questions and gotten 3 incorrect, which nets me a performance of 97%. My percentage above passing is reported as 50%, though it’s worth noting that the scale maxes out at plus/minus 50, which mathematically can’t be correct if the minimum passing threshold can be as low as 50%, so who knows. Anyway:

97 = X*1.5

X = 97/1.5 = 64.667

So we can estimate that I needed to get ~65% correct to meet the passing threshold given my question exposure, which is right in the middle of the current reported range.

Caveats include the fact that you don’t know how many if any of your questions are currently non-scoreable nor is it easy to see how many if any questions were subsequently nulled due to being deemed non-valid, so this really is just an estimate.

In related news, it’s lame there isn’t a way to see exactly how many questions you’ve answered and your raw performance at a glance. I assume they are trying to simplify the display to account for the varying difficulty of the questions, but it feels like obscuration.

Self-governance

The ABR has stated that the OLA question ratings from the community are all that’s used for scoring. While volunteers, trustees, etc are all enrolled in OLA and presumably rating questions, the ABR says what the greater community says is what goes.

So we are to judge what a radiologist should know based on what about 1/3 of MOC radiologists think a radiologist should know. There is no objective standard or aspirational goals. Radiologists can cherry-pick their highest-performing subspecialties and maintain their certification by answering a handful of questions in their favored subjects that are essentially vetted by their peers, who may not exactly be unbiased in the scoring process.

I don’t actually necessarily see a problem with that.

What I do see is more evidence of a bizarre double standard between what we expect from experienced board-certified radiologists and what we expect from trainees.

In court, the ABR has argued that initial certification and maintenance of certification are two aspects of the same thing and not two different products. As someone who has done both recently, they don’t feel the same.

Unlike OLA, neither residents nor the same broad radiology community has any say in what questions meet the minimal competence threshold for the Core and Certifying exams. For those exams, small committees decide what a radiologist must know.

You earn that initial certification and suddenly all of that mission-critical physics isn’t important. Crucial non-interpretive skills are presumed to last forever. And don’t forget that everyone should know how to interpret super niche exams like cardiac MRI (well, at least for two days).

It either matters or it doesn’t.

Why does physics, as tested by the ABR (aka not in real life), matter so much that it can single-handedly cause a Core exam failure for someone still in the cocoon of a training program but then suddenly not matter a lick when the buck stops with you?

The fact is that most practicing radiologists would not pass the Core Exam without spending a significant amount of time re-memorizing physics and nuclear medicine microdetails, and the new paradigm of longitudinal assessment doesn’t allow for once-per-decade cramming necessary to pass a broad recertification exam.

Meanwhile, diplomates are granted multiple question skips, which while designed to give MOC participants the ability to pass on questions that don’t mesh with their clinical practice can nonetheless be used to bypass questions that you don’t know the answer to. Especially for the performance-conscious and those with borderline scores, this creates a perverse incentive to miss out on rare learning opportunities.

But my point here is not that OLA should be harder.

My point is that certification should mean the same thing with the same standard no matter how long you’ve held it.