ABR simplifies Core Exam scoring

After years of pretending that people could actually fail (“condition”) individual exam sections other than physics in its convoluted two-stage exam scoring process, the ABR has decided to simplify things going starting this year in 2018.

From now on, there are three scoring outcomes:

  • PASS if you get a score of 350 or higher when averaging all sections together (and specifically pass physics)
  • CONDITION if you pass the overall exam but score less than 350 on physics
  • FAIL if your overall score is less than 350 when averaging all sections together

Conditioning physics means re-taking just physics. Failing means re-taking the whole thing.

This means that your performance on any individual section (except physics) is irrelevant so long as the average score across all sections meets the passing threshold of 350. No surprise there. For followers of last year’s mammography kerfluffle, you’ll remember that the ABR acknowledged that the results of the mammography section in isolation literally had no bearing on a single examinee’s passage result. Whether or not it was really technically possible to condition a non-physics section, no has ever conditioned a section other than physics since the Core Exam’s inception.

Scoring is still cloudy, however, because the passing threshold of 350 is a meaningless number without any measure of the preparation required or the percentage of questions you must answer correctly in order to achieve that score. It’s purportedly derived from the sum of the Angoff method scores for each section based on what the expert panel believes a “minimally competent” radiologist should know. So, whatever. This does mean, however, that strong sections can make up for weak sections. Consider this is your license to ignore GI and GU fluoroscopy.

While this sounds like a big positive development, I believe this is basically just a paper change. The ABR is just acknowledging outright the reality on the ground for the past several years: The large gap between overall passing performance and the true failure threshold for all non-physics sections is so large that in practice no one could actually fail an individual section.

Frankly, I wouldn’t be surprised if the one person per year who should have conditioned a non-physics section was just given a score of 200 on the offending set in order to pass via an informal secretive score floor. Who knows.

But at least it’s simpler and more straightforward now.

ABR manages expectations for the 2018 Core Exam

In the wake of last year’s impressive technical failure during the Core exam, the ABR has decided to try something new.

On Monday, when registration opened for the 2018 Core Exam, the ABR decided to not send all candidates the email at the same time. Instead, swathes of people had their invites delayed by several hours.

By the time these lucky folks received their invites, the Tucson test dates were completely filled (possibly because the Tucson experience is slightly nicer but maybe also because Chicago was the site of last year’s cluster). Additionally, the Chicago hotel block was completely booked for the first test dates.

This is an amazing illustration of managing expectations.

Yes, by screwing up something as easy and seemingly straightforward as sending an email literally as soon as possible during the testing process, the ABR has again angered a lot of people. But, but, they’ve also made sure to lower expectations in advance this year. Now, assuming they can administer the exam people have paid them for, everyone will just be pleasantly surprised that they can actually take the miserable two-day pain-fest from start to finish.

Clever.

 

 

Mammogeddon: Yes, the conclusion

This is my fourth (and final) post about the little snafu surrounding the mammography portion of the ABR Core Exam last summer.

  • I wrote about what happened here.
  • I wrote about what the response was here.
  • I wrote about the proposed solution here.

Now, we’ll finish with how that do-it-yourself online module went.

Logistically, it went great. By all accounts I’ve heard, people were able to log in from the comfort of whatever chair they were sitting at and take the module. The content was reportedly pretty much as expected for a Core exam mammo section, with the possible surprise for some of the inclusion of physics and non-interpretive skills (which are, after all, folded into every core exam section).

No surprise there, because as you might recall, ABR Executive Director Valerie P. Jackson had told examinees not to worry (emphasis mine):

The ABR has also heard from several residents who are concerned that they now need to completely re-study for the breast imaging module. As the ABR’s executive director, I (Dr. Jackson) personally reviewed the breast questions on the new module to modify any material that might not be visible on a monitor that is not high resolution. Although I am a breast radiologist, I have not practiced any clinical work or studied for an exam in more than three years. I found the content to be straightforward and inclusive of the important breast imaging concepts that candidates will most likely have retained from adequate initial exam preparation. Extensive re-study should not be necessary.

The invitation email went out July 27 and registration closed August 11. The module was offered on September 7 and 18, and the results were available on September 28.

As the make-up module was taken on the honor code, we’ll never know if anybody cheated, but it appears at least that no one was caught. On the plus side, we can applaud the ABR for not trying to install any spyware on examinees. Big brother was not invited to the party.

While the module took place several months after the usual pretest studying frenzy, reviewing the content for just one section, particularly mammography, was a stressful but probably not particularly tall order. I imagine nearly everyone took the section honestly.

Now, if you remember, the amusing part of the entire endeavor is that the ABR has admitted in the past that performance on the mammo module (or any individual section for that matter) essentially does not matter in terms of passing the test. No one has ever failed a single section other than physics in the years since the Core exam was first administered.

So, given several years of history to temper expectations, are the results of the module as expected? Did everyone pass?

Yes and yes.

I actually asked the ABR via email what the results were, and I got the impression that they did not want to tell me the specific truth because after a delay of about six weeks they gave me the default phrasing they love to use when discussing exam results:

In regards to specific details such as passes and fails for the breast category, the results for this breast imaging module were inline [sic] with the results from previous ABR exams.

…which means that everyone passed, which they later confirmed in a follow-up email.

I, for one, do you not understand the ABR’s reticence and coyness concerning discussing exam results. For example, while the results for the Core exam are more or less released annually, the results of the Certifying exam have never (to my knowledge) been disclosed publicly (e.g. see the official scoring and results page). One presumes that the most likely explanation is that the certifying exam pass rate is 100% and that the ABR is concerned people might question the necessity and utility of an exam with universal passage but that they also don’t want to make it hard and anger a bunch of practicing radiologists who are probably doing just fine.

But we’re not fooling anyone here. The issues with both initial certification and MOC are neither unique to radiology nor subtle. Transparency and accountability should be the sine qua non for a medical specialty board. And yet.

Surprises All Around

A VA branch is under investigation for poor quality radiology care (and for firing the whistleblower in retaliation):

As many as four to five times a day, Leskosky said, he found serious errors in prior readings, despite just four other radiologists being on staff. In one particularly egregious case, a radiologist missed a 17-centimeter tumor in a patient’s pelvis.

In private practice, radiologists may miss key findings once or twice in a lifetime, Leskosky said.

A large part of the problem, Leskosky said, is some of the other radiologists on staff were flipping through 50 to 60 patient scans a day, instead of the industry recommended 25 to 30 and, as a result, missing critical findings.

Losing a 17-cm tumor is a pretty aggressive miss, but 1) people in private practice absolutely miss a key finding more than once or twice per lifetime and 2) there is no “industry” to recommend a work-level (let alone one that’s used in practice).

Firing the whistleblower, however, is a pretty egregious no-no, and I’m pretty sure I’ve done some online modules at the VA about that being against the rules.

All said, the “industry” does need better PR though, because there are a lot of radiologists in practice who would love to read just 25 cases a day.

Mammogeddon 2017: The Conclusion?

From the ABR’s July 19 email:

Some of you are wondering why it has taken so long for the ABR to provide a solution. We apologize for the delay as we know this has been stressful for you. More than 450 candidates were affected by this situation. The cause of the problem was not initially apparent, and it was important for us to have time to investigate, review preliminary scores of all candidates, obtain direction from our board members and some program directors, and devise a solution that was most appropriate for all stakeholders, including you, your program, and your patients.

The ABR board also received input from the breast imaging community, which feels it is imperative for residents to be tested on breast content at some point in the certification process. The board members considered requiring a breast module on the Certifying Exam for those who did not receive the module on the Core Exam. However, all were concerned that more than two years of delay would require you to study again for the breast module.

The board feels strongly that we must administer the content as soon as possible, and that we should not require travel, other expenses, or additional resources, which is aligned with what we have heard from the breast imaging community. Therefore, we decided that we should trust you to take the online module in a setting of your choice. In addition, the breast module has been carefully edited to ensure that all findings are visible without the need for a high resolution monitor.

[…] We will schedule residents who need to take the breast imaging module at specific times on two dates: September 7 and September 18, 2017. You will select your desired start time when you register.

Still missing: what actually happened in Chicago, what the technical glitch was, how they’ve taken steps to prevent this from happening again, how this module is graded, how “hard” it will be to pass, if it’s actually possible to fail, and a finally—what happens if someone actually manages to fail.

It is interesting that you can take it anywhere you want but that you still must take it at specific times—presumably a compromise to prevent cheating/sharing of the exam content without resorting to using an official testing center. The real exam is proctored with a bathroom monitor, but the fabled mammo content is on the honor code. To me, this is highly suggestive of lip service to an apparently deeply hurt mammography community.

And, speaking of testing centers, the ABR recently released the following narrative about why they haven’t been able to disseminate the exam:

These delivery requirements have proven to be insurmountable obstacles for the numerous commercial testing vendors that we’ve engaged over the years. It’s important to remember that the vast majority of these vendors’ clients deliver text-based question exams with little or no multimedia content.

[…]

Just last year, we engaged two prominent commercial testing vendors to explore our goal of delivering the diagnostic radiology initial certification exams at local testing centers. Both vendors were given in-depth details of our exam delivery needs and asked to provide a proposal for our consideration […]

…but neither was interested.

I like that they’ve finally publically responded to these perennial requests.

I imagine these two were Prometric and Pearson VUE, because (despite the claim of “numerous” vendors) there are only a handful of large commercial testing centers around that could possibly furnish the exam. I suppose it’s possible the big two passed in years past. I have no doubt that the ABR’s demands for administration are not worth the time and expense for most vendors to meet given the low exam volume. The follow-up question, however, is whether or not it’s possible to write a Core exam that can be disseminated.

For example, the video portions of the exam are small in number and generally useless outside of cardiac MR (which, if we’re being honest, plays a comically outsized role on the test). The multi-slice scrolling capacity is rarely used and usually only a handful of images anyway. Mammo and radiographs could be selected that do not require high-resolution high-filesize images. The ACR in-service exam, of note, was able to snag a contract and is also image-based.

We are committed to making the initial certification process as facile as possible. While our past efforts have not been successful, we will continue to pursue our goal (and your wish) of delivering diagnostic radiology exams in local commercial testing centers. As we all know, technology is constantly evolving, and perhaps local exam delivery will become more feasible in the future.

I don’t doubt that the exam the ABR created couldn’t be ported to Prometric as is. Shucks, it didn’t even work in Chicago. But couldn’t we have a Core Exam that was functionally equivalent but wasn’t so off-putting? Exams need to be written with the administration in mind from the onset, not just as an afterthought.

Perhaps putting our hopes in the possibility that bandwidth and memory will be so cheap one day that testing companies won’t find our poorly written and conceived exam so unpalatable isn’t the best plan.