Step 1 Score Correlations

People often ask me about Step 1 corrections, particularly with regards to the Free 150 (for which I’ve posted explanations for several years). The data I’d come across over the years was super old.

Last month, Reddit user Waygzh posted the results of a 208 person survey (with an above average mean score of 245), which includes correlations for UWorld, the Free 150, multiple NBMEs etc.

The spreads are huge and the correlations not particularly good, but it’s the best you’re likely to get. Just don’t get discouraged if the number you see isn’t the number you want. Inspiration is better than deflation.

 

It was the best of exams. It was the worst of exams.

From the awesome and scathing “What Went Wrong With the ABR Examinations?” in JACR:

The new examination format also does a poor job of emulating how radiology needs to be practiced. Each candidate is alone in a cubicle, interacting strictly with a computer. There is no one to talk to and no opportunity to formulate a differential diagnosis, suggest additional imaging options, or provide suggestions for further patient management. The examination consists entirely of multiple-choice questions, a highly inauthentic form of assessment.

Only partially true. Questions can ask you for further management. Additionally, it’s possible to formulate questions (via checkbox) that allow you select reasonable inclusions for a differential. This isn’t the same as having a list memorized but is in some ways more accurate in the world of Google, StatDX etc. Of course, this kind of question isn’t meaningfully present, but MOC doesn’t preclude certain lines of testing.

Another rationale for the new examination regimen was integrity. Yet instead of reducing candidate reliance on recalled examination material, the new regimen has increased it, spawning at least six commercial online question bank sites. The fact that one of the most widely used print examination preparation resources is pseudonymously authored is a powerful indicator that the integrity of the examination process has been undermined, effectively institutionalizing mendacity.

Every board exam has qbank products. Part of why Crack the Core is pseudonymously authored isn’t just the recalls; it’s presumably also related to his amusing but completely unprofessional teaching style. I very much doubt the Core Exam is more “recalled” than prior exams. What we should be doing is acknowledging that any standardized test will be prepared for this way via facsimile questions, and there is literally no way to avoid it. It’s not as though Step 1 is any different.

Many of the residents we speak with regard the core examination not as a legitimate assessment of their ability to practice radiology but as a test of arcana. When we recently asked a third-year resident hunkered down over a book what he was studying, he replied, “Not studying radiology, that’s for sure. I am studying multiple-choice tests.” The fact that this sentiment has become so widespread should give pause to anyone concerned about the future of the field.

Yes, this is true. But it also strikes me that the old school boards wrapped a useful and worthwhile skill in a bunch of gamesmanship, BS, and pomp. Nonetheless I can’t dispute that casemanship skills have real-world parallels and that the loss of them may have resulted in some young radiologists sounding like idiots when describing a novel case in front of a group of their peers.

In essence, the ABR jettisoned a highly effective oral board examination that did a superb job of preparing candidates for the real-world practice of radiology and replaced it with an examination that merely magnifies the defects of the old physics and written examinations. The emphasis is now on memorization rather than real-time interaction and problem solving. In our judgment, candidates are becoming less well prepared to practice radiology.

It seems increasingly true that anyone more than a couple years of residency has now fully fetishized the oral boards. It’s definitely true that traditional case taking skills have rapidly atrophied. Residency may feel long but institutional memory is short. Old school casemanship isn’t really the same thing as talking to clinicians, but it certainly has more in common with that than selecting a best answer from a list of choices.

It is an important skill/ability to succinctly and correctly describe a finding and its associated diagnosis. Some residents now are still able to get the diagnosis but may struggle with describing the findings appropriately when on the spot. I don’t how much that matters in the long term. I would be interested in seeing if any of the old vs new debate has would have any impact on the quality of written reports, the fundamental currency of our field in the 21st century. I’ve seen plenty of terrible reports and unclear language from older radiologists, so the oral boards barrier couldn’t have been that formative.

The fact is that neither exam is a good (or even reasonable) metric. Frankly, a closed-book exam in and of itself is inherently unrealistic from daily practice. But any exam that trades in antiquated “Aunt Minnies” or relies on demonstrating “common pathology in unusual ways” are really dealing in academic mind games and not really testing baseline radiologic competence.

The movement to end Step 2 CS

If you hadn’t heard, there is growing movement to end Step 2 CS (because it’s a stupid, expensive, and ultimately ineffective test). You can read about the background and sign the petition here. There’s also a fun additional JAMA editorial.

  • 20,190 MD (ignoring DOs who mostly don’t take it and IMGs, for whom the test was originally designed) students took the test in 2014-15, of which 96% passed. So 807 failed.
  • 817 MDs took a repeat and 86% passed (presumably 10 of these were third attempts or re-attempts from the previous year).
  • So 114 US MDs were caught by the Step 2 CS hurdle, at the maximum.

So that’s a terrible value proposition: offloading an expensive test offered in a handful of locations to students drowning in debt and short on time in order to catch a relative handful of people in a deficiency that is largely contrived. But what happened to those 114, of which half failed for communication skills and half failed for poor [fake] “clinical” skills? How many students are actually prevented from continuing their careers? And for students that fail and then pass (the vast majority), is there any evidence whatsoever that this process has improved their skills?1

I am very curious about the former question. I strongly suspect the latter is completely absent.

The irony is that there are plenty of bad physicians, but none of this testing is well suited to unmasking and dealing with real world deficiencies. The even sadder wrinkle is that there are also clearly physicians in the US who have insufficient English skills to practice medicine properly, so Step 2 CS isn’t even doing what it was originally designed to do well.

If I were an MS1, I’d be praying this momentum snowballs and I could save myself the hassle and additional debt.


  1. This would also apply to the other Steps of course, but CS stands out as being more subjective, less predictable, and thus more frustrating for the students who fail.

Guide to The Core Exam

Below are the current entries in the ABR Core Exam Series:

The Core Exam Experience
Detailed post describing the process of registering for the exam, selecting testing location and lodging, the actual test day experience, and the exam itself including the the exam breakdown, scoring, and results

Approaching the Core Exam
Detailed post discussing approaching the exam, study schedules, picking resources, and exam content including physics & non-interpretive skills

Resources for the Core Exam
Breakdown and brief reviews of commonly used Core resources

Core Exam Question Banks Review
Separate reviews of the currently available Core Exam qbanks

Book Review: Crack the Core, Radiologic Physics War Machine, CTC Case Companion
Unnecessarily in-depth reviews of Prometheus Lionhart’s Crack the Core book series

Book Review: Core Radiology
TL;DR – I liked it

Yes, you can switch back from REPAYE to IBR or PAYE

There has been a lot of confusion from borrowers whether or not REPAYE, with its partial interest subsidy, is a good choice for people with high future income (e.g. residents). The main concern is what happens after training when salaries increase and the possibility of breaking past the monthly payment cap, which could make you lose money (in the context of trying to minimize payments in anticipation of PSLF). Note: If you’re just trying to pay off your loans in an efficient way, breaking past the cap should be mostly irrelevant–you should be trying to pay down your loans as fast as possible anyway.1

If you call your federal loan servicer but don’t ask the right questions, your servicer may lead you astray in how they answer questions about the terms of the REPAYE program. It’s misleading but technically true: if you are making so much money that you break past the REPAYE cap, you absolutely cannot switch back to PAYE or IBR.

That’s NOT because you aren’t allowed to switch out of REPAYE in general (you are), but because at that point you would no longer have a “partial financial hardship” and thus no longer qualify for those plans to begin with. Your servicer is able to provide information and advice, but don’t for a second think that they don’t have a vested interest (see what I did there?) in your payments. A simple rule of thumb is that if you owe more on your loans than you make in a year, you definitely still quality for your income driven repayment plan.

What is actually used for payment calculations is not your gross income but your discretionary income: your gross income minus 150% of the federal poverty line for your family size (e.g. family size of 1, 2, and 3 is &17,655, $23,895, and $30,135 in 2015, respectively). The official rule is that if your calculated monthly PAYE/IBR payment (whichever you qualify for) using 10/15% of your discretionary income is less than the standard 10-year repayment, then you still qualify.

So there is a simple solution for forward-thinking borrowers who want to take advantage of the REPAYE benefits but don’t want to tie themselves to higher future payments: Switch back before you make money.

You can switch from REPAYE to PAYE as long as you still qualify for PAYE. Or you can switch back to IBR instead if you had older loans and didn’t qualify for PAYE to begin with.2 Do this at the end of your training and the problem is solved. (Technically, many people could do it even once out in practice; it all depends on how much you borrowed versus how much you/your family makes per year. You can use the calculator to see what household income you’ll need to break past the threshold.)

Also note that since most people generally use tax-returns and not pay stubs to verify income, there is generally a delay between when your income rises and when your taxes reflect that increase. This isn’t the way servicers would like it, but it’s the reality on the ground. You could be an attending as of July 2016, but when you resubmit income verification in the fall of 2016 for REPAYE, you’ll be submitting your 2015 taxes, which is a combination of your last two PGY years of training.

The bottom line is that you absolutely can switch out of REPAYE—you just have to be a little bit thoughtful on when you want to switch out to not miss the window. REPAYE makes the most sense for many if not most residents. For people who aren’t going for PSLF (especially if they’ve borrowed smaller amounts and won’t enjoy a big interest subsidy), no-cost private refinancing may be a better choice.

This plan-switch information comes from this document and FAQ, and I’ve confirmed this interpretation with Nelnet (one of the federal loan servicers). If you talk to your servicer and they say otherwise, ask them to explain exactly why and we’ll get to the bottom of it. Because they should be wrong.

 


  1. I.e. it would mostly likely only matter if lifestyle has inflated to the point where the money you should be using to pay down your loans has been earmarked for other purposes, like car loans.

  2. IBR payments are higher than PAYE payments (15% vs 10% of AGI), so PAYE is preferable in the context of minimizing payments for PSLF.