Radiology Quality Police

04.21.25 // Radiology

What is quality care, and how do you define it?

I suspect for most people that quality is like pornography in the classic Supreme Court sense—you know it when you see it. But quality is almost assuredly not viewed that way when zoomed out to look at care delivery in a broad, collective sense. Instead, it’s often reduced to meeting a handful of predetermined outcome or compliance metrics, like pneumonia readmission rates or those markers of a job-well-done as defined in the MIPS program.

The reality is that authoritative, top-down central planning in something as variable and complicated as healthcare is extremely challenging, even if well-intentioned. As Goodhart’s law says, “When a measure becomes a target, it ceases to be a good measure.”

In the real world, I would argue a quality radiology report is one that is accurate in its interpretation and clear in its communication. But without universal peer review, double reading, or an AI overlord monitoring everyone’s output, there is no way to actually assess real quality at scale. You can’t even tell from the report if someone is correct in what they’re saying without looking at the pictures. Even “objectively” assessing whether they are clear or helpful in just their written communication requires either a human reviewer or AI to grade language and organization by some sort of mutually agreed-on rubric. It’s simply not feasible without a significant change to how healthcare is practiced.

And so, we resort to proxy metrics—like whether the appropriate follow-up recommendation for a handful of incidental findings was made. The irony, of course, is that many of these quality metrics are a combination of consensus guidelines and healthcare gamesmanship developed by non-impartial participants with no proof they reflect or are even associated with meaningful quality at all.

We should all want the quality of radiology reporting to improve, both in accuracy and in clarity. Many of these problems have been intractable because potential solutions are not scalable with current tools and current manpower—which is why soon you’ll be hearing about AI for everything, because AI solves the scaling problem, and even imperfect tools over the coming years will rapidly eclipse our current methods like cursory peer review.

Everyone would rather have automated incidental finding tracking than what most of us are still using for MIPS compliance. Right now, it’s still easy to get dinged and lose real money because you or your colleagues omitted some BS footer bloat about the source of your follow-up recommendations for pulmonary nodules too often. Increased quality without increased effort is hard to complain about.

But even just imagine you have a cheap LLM-derived tool that catches sidedness errors (e.g. right abnormality in the findings and left in the impression) or missing clarity words like forgetting the word “No” in the impression (or, hey, even just the phrase “correlate clinically”). This already exists: it’s trivial, requires zero pixel-based AI, (and—I know—is rapidly becoming table stakes for updated dictation software), but widespread adoption would likely have a more meaningful impact on real report quality than most of the box checking we do currently. A company could easily create a wrapper for one of several current commercial products and sell it tomorrow for those of us stuck on legacy systems. It might even be purchased and run by third parties (hospitals, payors, Covera Health, whatever) to decide which groups have “better” radiologists.

But now, take it that one step further. We’ve all gotten phone calls from clinicians asking us to translate a colleague’s confusing report. Would a bad “clarity score” get some radiologists to start dictating comprehensible reports?

It’s not hard to leap from an obviously good idea (catching dictation slips) to more dramatic oversight (official grammar police).

Changes to information processing and development costs mean the gap between notion and execution is narrowing. As scalable solutions proliferate, the question then becomes: who will be the radiology quality police, and who is going to pay for it?

Rational Alchemy

04.16.25 // Miscellany

From the excellent Alchemy: The Dark Art and Curious Science of Creating Magic in Brands, Business, and Life by Rory Sutherland:

In theory, you can’t be too logical, but in practice, you can. Yet we never seem to believe that it is possible for logical solutions to fail. After all, if it makes sense, how can it possibly be wrong?

[…]

If you are a technocrat, you’ll generally have achieved your status by explaining things in reverse; the plausible post-rationalisation is the stock-in-trade of the commentariat. Unfortunately, it is difficult for such people to avoid the trap of assuming that the same skills that can explain the past can be used to predict the future.

The world trades in stories, but compelling stories aren’t necessarily true. See also: hindsight bias. The post hoc seeming inevitability of where we are now is a mirage.

Adam Smith, the father of economics – but also, in a way, the father of behavioural economics – clearly spotted this fallacy over two centuries ago. He warned against the ‘man of system’, who: ‘is apt to be very wise in his own conceit; and is often so enamoured with the supposed beauty of his own ideal plan of government, that he cannot suffer the smallest deviation from any part of it. He goes on to establish it completely and in all its parts, without any regard either to the great interests, or to the strong prejudices which may oppose it . . . He seems to imagine that he can arrange the different members of a great society with as much ease as the hand arranges the different pieces upon a chess-board. He does not consider that the pieces upon the chess-board have no other principle of motion besides that which the hand impresses upon them; but that, in the great chess-board of human society, every single piece has a principle of motion of its own, altogether different from that which the legislature might chuse [sic] to impress upon it. If those two principles coincide and act in the same direction, the game of human society will go on easily and harmoniously, and is very likely to be happy and successful. If they are opposite or different, the game will go on miserably, and the society must be at all times in the highest degree of disorder.’

This chess piece argument is also a metaphor favored by conservative economist Thomas Sowell as the key failing conceit of central planning.

The problem that bedevils organisations once they reach a certain size is that narrow, conventional logic is the natural mode of thinking for the risk-averse bureaucrat or executive. There is a simple reason for this: you can never be fired for being logical. If your reasoning is sound and unimaginative, even if you fail, it is unlikely you will attract much blame. It is much easier to be fired for being illogical than it is for being unimaginative.

[…]

The fatal issue is that logic always gets you to exactly the same place as your competitors.

The primary function of managers is to preserve their position within management. The second function is to be promoted. The distant third is to actually manage people well or improve their organizations. (Further Reading: Academic Medicine and the Peter Principle)

The late David Ogilvy, one of the greats of the American advertising industry and the founder of the company I work for, apparently once said, ‘The trouble with market research is that people don’t think what they feel, they don’t say what they think, and they don’t do what they say.’*

[…]

It is fine to provide up-to-date magazines in reception to show that you care, but when the urge to show commitment to patients involves performing unnecessary tests and invasive surgery, it probably needs to be reined back.

Yes. I am reminded of Patient Satisfaction: A Danger to be Avoided.

If you want to change people’s behaviour, listening to their rational explanation for their behaviour may be misleading, because it isn’t ‘the real why’. This means that attempting to change behaviour through rational argument may be ineffective, and even counterproductive. There are many spheres of human action in which reason plays a very small part. Understanding the unconscious obstacle to a new behaviour and then removing it, or else creating a new context for a decision, will generally work much more effectively.

Behavior change is hard. I can barely control a whole host of my own impulses, let alone guide others.

The self-regarding delusions of people in high-status professions lie behind much of this denial of unconscious motivation. Would you prefer to think of yourself as a medical scientist pushing the frontiers of human knowledge, or as a kind of modern-day fortune teller, doling out soothing remedies to worried patients? A modern doctor is both of these things, though is probably employed more for the latter than the former. Even if no one – patient or doctor – wants to believe this, it will be hard to understand and improve the provision of medical care unless we sometimes acknowledge it.

[…]

To put it crudely, when you multiply bullshit with bullshit, you don’t get a bit more bullshit – you get bullshit squared.

[…]

Nassim Nicholas Taleb applies this rule to choosing a doctor: you don’t want the smooth, silver-haired patrician who looks straight out of central casting – you want his slightly overweight, less patrician but equally senior colleague in the ill-fitting suit. The former has become successful partly as a result of his appearance, the latter despite it.

The Taleb reference is commonly referenced even if it may not always work in real life. It’s one of those lightbulb-generating remarks that strikes that magic of being surprisingly intuitive after seeming counterintuitive.

There is an important corollary: proxy metrics (patrician manner, various training factors) don’t actually mean what we want them to mean. You want to see a Harvard-trained doctor because you assume they are better through the implied meritocratic scarcity of an elite institution and the presumption that therefore, surrounded by other geniuses and some presumably fancy digs, their training was uniquely better—but there is no actual basis for this belief.

In making decisions, we should at times be wary of paying too much attention to numerical metrics. When buying a house, numbers (such as number of rooms, floor space or journey time to work) are easy to compare, and tend to monopolise our attention. Architectural quality does not have a numerical score, and tends to sink lower in our priorities as a result, but there is no reason to assume that something is more important just because it is numerically expressible.

Measurability does not equal importance. See “Overweighing what can be counted” in Munger’s Incorrect Approaches to Medicine.

The more data you have, the easier it is to find support for some spurious, self-serving narrative. The profusion of data in future will not settle arguments: it will make them worse.

…I naively thought Covid would bring people together. And it did, for maybe a week or two. Then the dueling data wars began.

We are flush with data. Absolutely awash in data. If the past few years of social media have taught us anything, it’s that information isn’t truth. It’s raw material for storytelling. Yoval Noah Harari does a nice if depressing job discussing information networks in his recent book, Nexus.

The toddler puts on a show of having an argument, but they are holding a tantrum in reverse. If they ‘win’ the argument, no tantrum is needed. If they lose, they can tell themselves that they tried but the other person deserved the tantrum because they didn’t listen.

– Seth Godin, “How to win an argument with a toddler” (P.S. You can’t)

// 04.15.25

Who or What is the O-Ring? And Who or What Will It Become Over The Coming Years?

04.14.25 // Medicine, Miscellany

From Stripe’s 2024 Annual Letter:

Much as SaaS started horizontal and then went vertical (first Salesforce and then Toast), we’re seeing a similar dynamic playing out in Al: we started with ChatGPT, but are now seeing a proliferation of industry-specific tools. Some people have called these startups “LLM wrappers”; those people are missing the point.

The O-ring model in economics shows that in a process with interdependent tasks, the overall output or productivity is limited by the least effective component, not just in terms of cost but in the success of the entire system. In a similar vein, we see these new industry-specific Al tools as ensuring that individual industries can properly realize the economic impact of LLMs, and that the contextual, data, and workflow integration will prove enduringly valuable.

The “O-ring” of economist Michael Kremer’s original 1993 paper is a reference to the tiny failure point that caused the 1986 Challenger space shuttle disaster. Without diving into obscure references, we could summarize: a chain is only as strong as its weakest link, and, as a corollary: in workflows with multiple interdependent steps, errors can multiply.

The better we as humans perform and the better our systems become, the more each smaller failure begins to dominate the pain and inefficiency landscape.

Stripe is a massive payments processor and works with a large fraction of the world’s companies, and I suspect they are right.

Many have argued that AI models are likely to become the utilities that power new products and less likely to become the dominant products themselves. That remains to be seen, of course, but at least with current LLMs, some specialized wrapping is necessary to make these tools function in high-stakes, specific environments that have real work products outside of making a generic knowledge worker generically more productive or generate some fun art.

Healthcare, in particular, complicates this by adding in a long enterprise sales cycle and layers of slow human bureaucracy.

We’ll have to see what kind of durable moat—if any—any player from this nascent stage has. Capabilities are going up and training costs are going down, so even the simple question of how much wrapping you need to successfully deploy new tools is an impossible question given the unstable, constantly shifting sands at the frontiers of AI. With enough compute and time, perhaps dominant frontier players like OpenAI and Anthropic will simply make models that are robust enough to basically learn and do anything.

Stripe’s argument is that, like a plug adapter, we need those wrappers to solve the O-ring problems for implementing new products and improving processes. As a payments processor, it’s clear why Stripe would want to see a world where lots of companies make great businesses serving lots of other companies.

I’m not sure the current state of the art is useful for making longer-term predictions. Will we see a few novel foundational healthcare models as total farm-to-table solutions? Everyone perhaps choosing from a buffet of numerous efficient, cheaper, smaller narrower models a la carte from a handful of marketplace aggregators, like some of the nascent players have made possible? A few dominant models (e.g. OpenAI) but with numerous wrappers and customers mostly choosing between different implementations (largely ignorant of the machinery under the hood)? Or just perhaps one or two dominant models that are able to be or make everything for everyone? Understanding how long the wrapping stage of AI deployment lasts is likely both of function of how optimistic you are about hyperscaling and also a bunch of idiosyncratic industry, regulatory, and human factors for whatever use-case you care about.

Is wrapping a durable play or just a temporary necessity?

Radiology Surge Pricing

04.09.25 // Radiology

As we discussed in The Necessity of Internal Moonlighting, you can regularly need some extra manpower to maintain turnaround times or mitigate misery without the need for a full additional FTE shift on the schedule (or, alternatively, where you do need some real shiftwork but don’t want to press people into service without additional reward).

Take this recent article about “Surge” staffing in radiology as described in Radiology Business:

On-service radiologists utilize Microsoft Teams to contact available nonscheduled rads during periods of heavy demand. Team members who are available then can log on remotely and restore the worklist to a “more manageable length,” logging their surge times in the scheduling system in five-minute increments. Compensation is based on the duration of the surge and time of day when it occurs.

Just-in-time overflow help is an important use case of internal moonlighting, and doing this with less friction is exactly what LnQ is trying to facilitate and streamline.

Giving advice and selling can’t be the same thing.

Nassim Taleb, pithily summarizing a lot of problems. For example, the core problem of much of the financial planning industry.

// 04.08.25

Dealing with an ABR Core Exam Failure

04.07.25 // Radiology

Firstly, it should almost go without saying, but: you can do this.

I’d also like to acknowledge that nothing below is particularly noteworthy or novel advice. The Core Exam is like the other high-stakes multiple choice exams you’ve taken except for the fact that it has more pictures.

And, of course, the question of how to pass the Core Exam after a failure is mostly the same as asking how to pass it in the first place. Before we get further, I published a series of articles grouped here as a “Guide to the Core Exam” that lays out a lot of helpful information. There are some out-of-date passages about failing because physics and in-person details, but the core information is unchanged.

Acknowledge Luck

As you no doubt noticed during your first attempt(s), the questions on the ABR exams are somewhat arbitrary in what they include, so part of your performance is luck of the draw. While the difficulty is curated, the specific content breakdown is not a perfect cross section of possible topics. You can have the same diagnosis multiple times but then zero questions on broad swaths of important material. How your knowledge gaps line up can make a big difference.

Your performance on a given day is a product of your variable skill (energy, focus, attention, etc) and the exact exam you get. All things being equal, that means that a borderline failure is also truly a near pass.

Dissect Your Performance

Look at the two breakdowns: organ (breast, cardiovascular, GI, GU, MSK, neuro, peds, and thoracic) and modality (CT, IR, MR, Nucs, radiography, US, NIS, and Physics). Find if you have outliers, and plan to shore up those shortfalls with extra dedicated targeted review.

At the same time, do not neglect your strengths entirely. Backsliding is counterproductive.

The nature of spaced repetition is that you need more reps more frequently for new knowledge and problem areas and fewer reps spaced further apart for your strengths—but you still need reps across the board.

Further Reading: Learning & The Transfer Problem

Review Your Study Methods

What exactly was your study method and plan for your initial attempt(s)?

There are a couple of maladaptive tendencies common amongst medical students that can persist into residency:

The tendency to graze across too many resources. Focus on fewer things and learn them deeply.
The tendency to spend too much time passively reading (and especially re-reading) books like Crack the Core at the expense of doing lots of high-quality questions. We are radiologists, and the majority of the exam is image identification: you need to look at lots and lots of pictures.

When it comes to doing practice questions, you also need zoom out and look for trends:

More than just stronger/weaker subspecialty performance, are there any themes to why you get questions wrong? Is there a time component? Is it that you often don’t see the finding in the picture? That you simply don’t know the diagnosis? Or that you’re being fooled by differential considerations and need to focus on key features that distinguish between plausible alternatives? Is it a confidence issue and you’re overthinking it, getting spooked by questions that seem too easy? If you change answers, are you more likely to change from wrong to right or right to wrong? (I think most people overthink it and change for the worse).

If there’s a pattern, it can be the key to unlocking performance.

Further Reading: Dealing with Test Anxiety and Demoralization

Questions/Cases > Reading >> Re-reading

First: Horses > Zebras.

In general, the biggest bang for your buck is still going to be common diagnoses (including weird looks of common things) and normal variants over esoterica. Rare things arise most when they are absolute Aunt Minnies that you can tell at a glance (hence the need for lots of image-based questions).

On a related note, if you never saw them, the ancient free official ABR practice test from 2014 is available on the Wayback machine here.

Also worth mentioning: NIS is a bunch of random material. Many people can read the manual a copy of times and get the job done here, but the reality is that these need to mostly be easy points. If you don’t retain pseudoscientific business jargon naturally, then don’t shirk the review here. The NIS App, for example, is well-reviewed, but there is also an Anki deck as well.

Spaced Repetition

You can use the ACR DXIT/in-service Anki deck for a large number of good free questions. You could also use one of the massive Core Radiology decks. But for the second round of studying after a failure, making quick cards of every question you guess or get wrong from whatever source you’re using with your phone’s camera or with screenshots and incorporating that into repeated longitudinal review may be the highest yield.

In Windows, for example, Windows+Shift+S opens up a quick adjustable screenshot reticle that will copy that portion of your screen to the clipboard.

On Mac, the adjustable screenshot shortcut is Shift+Command+4, which automatically saves to desktop. To save to your clipboard, add Control, so Ctrl+Shift+Command+4.

The Reality

Passing is almost certainly a matter of doing more, high-quality reps while not over-focusing on weaknesses such that you backslide on your relative strengths. The Core Exam is arbitrary enough that some of your high and low-performance areas may not be as meaningful as you think, so you need to continue broad reps in addition to the extra targeted review.

Once you can emotionally recover from the setback and just get back to it, it’s going to work out.

Radiology Staffing Shortages and Group Culture

04.03.25 // Radiology

One thing our recent discussions of the nationwide shortage of radiologists didn’t include (in addition to a solution) is how the shortage has impacted the culture of radiology:

Pulse and a License

One of the concerning features of the current staffing shortage is the desperation with which many universities/hospitals/groups are recruiting new talent. When the market is tight and groups are well-staffed, groups get to be picky about cultural fit. Groups prioritizing compensation or efficiency can recruit fast radiologists or those with a broad skillset and flexible attitude. Groups that prioritize quality, academic productivity, or relationship-building can at least attempt to select for proxy features. Groups can grow in ways that align with their culture and mission.

But when there is far too much work to do and not enough people to do it, priorities shift to things like having a pulse and an active medical license, hopefully without board complaints or a felony record.

In Jim Collins’ Good to Great, there is a whole chapter dedicated to hiring. He argues that the key to creating a great business is not the mission or values, a charismatic leader, technology, or a clever strategic plan. The key foundation on which all other components rest is having the right people.

Sabotaging culture to generate revenues or get the work done may feel like a necessary choice in the short term (it also may be unavoidable when the alternative is operational insolvency), but it also has long-term consequences. Culture isn’t something you speak to; it is something that arises organically from good people doing things they believe in. As Collins says, “The best people don’t need to be managed.”

Fast Readers vs Practice Builders

One of the ways that cultural breakdowns manifest in radiology practices is the conflict between fast readers and practice builders (or, in academics: gold-star earners vs worker bees). Obviously, there are efficient radiologists who can do everything well, churning out an incredible volume of high-quality reports, and there are lazy people who do a bad job producing a relative molehill. Real life is a continuum, but the fake dichotomy is helpful for discussion.

I’m not even going to pretend that one is good or bad. The reality is that each of us is on a continuous spectrum as opposed to a caricature at the ends of the curve. The problem is that both extremes are real approaches and that good people in each camp become frustrated when the culture of the group does not align with their personal preferences. I see this discussed online all the time. Fast readers bemoan the freeloaders who are being paid the same or a similar amount for “less work.” But practices also rely on good citizens to get the important but non-remunerative work done. As an organization scales—and many groups have grown significantly over the past decade—reconciling these competing visions for an ideal radiology practice can seem impossible.

From a practice competition standpoint, it’s easy for a group to fall into a no man’s land where the compensation plan doesn’t reward speed enough for the high-productivity readers to be happy or rewards speed too much for the less “productive” members who get bogged down in the most complex cases, want more time to produce helpfully detailed reports, speak to clinicians, answer technologist questions, or teach residents. This can be compounded to disastrous effect by the cherry-picking that ravages some practices utilizing a productivity model, especially those that do not enforce list hygiene through automatic case assignment or customized internal work units to balance case desirability. If you log into a list and it’s full of plain films, thyroid ultrasounds, and chest CTs, you are logging into an environment where this hasn’t been figured out yet.

We touched on this previously in quality, speed, and “productivity”—and I didn’t have a magic bullet in that discussion either. These are real problems, and if there was a universal easy solution, everyone would be doing it. My feeling, as concluded in that brief article, is that the table stakes in the future are to automate case assignment ± customized internal RVUs to better account for effort ± account for “work” RVUs for nonbillable tasks (but bean counting every single thing you do has its own very real negative consequences). The quality/speed tradeoff is inherent to radiology work, but a productivity model that doesn’t take some variation of this into account is too ripe for cheating and gamesmanship.

Culture isn’t Optional; It’s Organic

My argument with the first two sections of this post is that compromising on hiring and being passive with internal work divisions and the productivity question are a one-two punch. I increasingly believe that places that don’t figure this out become revolving doors. At that point, recruitment is purely mercenary based on measurables. There are people who are willing to work that way, but long term, I don’t believe that leads to satisfaction or stability.

Culture happens whether we want it to or not, and creating a job that people enjoy and are loyal to doesn’t happen by accident. We are in a period of increased volatility in the radiology workforce regardless of what a practice does, but any job can become more stable if it feels like a meaningful career.

Will Automation Bias be Tractable?

03.31.25 // Radiology

In this previous post about breast imaging, we briefly touched on the soon-to-be-growing-and-maybe-even-critical problem of automation bias in radiology caused by the growing use of AI.

This study evaluating AI-assisted detection of cerebral aneurysms had similar findings:

Results
False-positive AI results led to significantly higher suspicion of aneurysm findings (p = 0.01). Inexperienced readers further recommended significantly more intense follow-up examinations when presented with false-positive AI findings (p = 0.005). Reading times were significantly shorter with AI assistance in inexperienced (164.1 vs 228.2 s; p < 0.001), moderately experienced (126.2 vs 156.5 s; p < 0.009), and very experienced (117.9 vs 153.5 s; p < 0.001) readers alike.

Conclusion
Our results demonstrate the susceptibility of radiology readers to automation bias in detecting cerebral aneurysms in TOF-MRA studies when encountering false-positive AI findings. While AI systems for cerebral aneurysm detection can provide benefits, challenges in human–AI interaction need to be mitigated to ensure safe and effective adoption.

Everyone got faster, but inexperienced readers were fooled by false positives.

This is going to be such a problem.

The reality is that using AI to make us faster is so incredibly ripe for these outcomes. Sure, we could using AI to catch mistakes after an initial independent rad interpretation, and then we could even set up such a system to then use a third party to adjudicate persistent disagreements in a blinded fashion (i.e. a neutral third party radiologist or maybe a different AI agent picks the winner without knowing who they side with)—but the raw economics absolutely point to us using AI as a resident first draft as soon as feasible. It’s going to get messy.

There is an argument that you will have to increasingly be an expert in order to outperform an increasingly competent algorithm. While many current machine mistakes are obvious to experienced radiologists, failures won’t always be comically clear in the future. Assuming we need humans for the long term, training and training quality are critical, and doing so in a way that shields humans from tainting and overreliance on computers will be key.

Yes, pilots use autopilot, but some of those big life-saving stories make the news precisely because pilots also sometimes need to take control.

My internet friends over at Medality are having a big Physician’s Week Spring Sale through March 31 (trainee extra discount link here). Solid use of CME funds before the end of the year + an easy way to support this site.

// 03.29.25