Backseat Epic

01.30.26 // Medicine, Miscellany

Mike Swanson’s “Backseat Software” perfectly describes one of the many modern software problems that also happen to plague the Epic EHR:

So the problem isn’t that software ever teaches, asks, or informs. The problem is that once a company builds the machinery to do it, that machinery becomes cheap to reuse, and the incentives gradually pull it away from “help the user succeed” toward “move the metric.”

What starts as an occasional heads-up becomes a permanent layer of UI exhaust.

And then:

So why does it keep happening? Because inside companies, the incentives are clear and the measurements are easy. You can measure clicks and track whether they led to a “completion.” You can measure whether a nudge led to the next step in the funnel.

You cannot easily measure the resentment. Or the rage clicks when they smash a button to dismiss another “did you know” pop-up. You cannot easily chart the moment a user thinks, “I used to like this product, and now it feels needy.” You cannot easily quantify the slow erosion of trust.

I have been dismissing the same feature tour prompt at every login for 5 years.

The Mediocrity of Scale

01.29.26 // Medicine

Different doesn’t mean right or wrong. Best practices are sometimes best, and sometimes they’re just a mediocre consensus that a small group agrees to.

Institutions are so often large and stupid because herds of humans are difficult to manage at scale, and the lessons learned by individuals often don’t become institutionalized without significant baggage in the form of procedural debt. So many individual actions and the proliferation of useless bureaucracy can be ascribed to the pursuit of plausible deniability. The fewer things that can be pinned on us, the safer our jobs. No one is fired for being rational, even if the outcome is suboptimal.

It’s not the goals but rather the incentives created by those goals that shape outcomes. It’s the BS that sounds good and reasonable in isolation versus what works in practice. The most important tactic for achieving any strategy is removing the friction, roadblocks, and unintended negative consequences of doing the actual desired behavior.

There is no broadly acceptable present value function to gauge political acts the way we can estimate the effects of economic decisions in terms of monetary value. The time horizon of politics, social media, and fame are all much shorter than the downstream consequences of most decisions, which incentivizes leaders of all stripes to do stupid shit that looks good now—even if it is, on the whole, harmful.

This is a problem because we often conflate “rational” behavior with optimization of easily measurable outcomes.

The creation of red tape is understandable, but the incremental costs are almost never considered and are challenging to undo, as every policy is obviously there for a reason—even if, on balance, that reason does not justify the cost. Nothing is easier than thinking of more things we can do to make something better or safer. What is harder is asking the tough question: at what cost? What are we willing to forgo in order to achieve this addition? Intentions matter, but the consequences matter more.

Bigger often means more inbound data and more layers of management to deal with it. Size isn’t always good, and growth shouldn’t always be the goal. Yes, size can help you fight a bully, and it can also help you be a bully. And there are efficiencies of scale when it comes to both high fixed costs as well as volume-based discounts on costs like IT. But we would be foolish to only look at those measurable costs and lose sight of the downsides: increasing bureaucracy, weak culture, lack of accountability, and a diluted mission.

Step into a large hospital and tell me if you think everybody feels like they’re on the same team. The fragmented care in your average mega-hospital is so often removed from a well-oiled machine. These organizations are essentially too large to have a mission or a really strong, behavior-shaping positive culture. Decision makers and rank-and-file are often separated by helplessness-inducing layers of human flotsam.

The mission of a large healthcare organization is not actully patient care. Healthcare is the industry, and patients are the vehicle for the business model. They are the sine qua non of healthcare profit. But the proliferation of bureaucrats and policies because of the need to “manage people” tells us that it’s not working.

If people were really working together, we wouldn’t need so much bloat.

Amazon founder Jeff Bezos on his decision to start an online bookstore in the 1990s:

The framework I found which made the decision incredibly easy was what I called the regret minimization framework. I wanted to project myself forward to age 80 and look back on my life and I want to have minimized the number of regrets I have. And I knew that when I was 80 I was not going to regret having tried this. I was not going to regret trying to participate in this thing called the internet that I thought was going to be a really big deal. But I knew the one thing I might regret is not ever having tried. And I knew that that would haunt me every day. So when I thought about it that way it was an incredibly easy decision.

With unavoidable uncertainty and wild problems, sometimes regret minimization is the best paradigm we have.

// 01.28.26

Basecamp’s Jason Fried’s brief discussion of product development should be required reading for anyone who designs software, including EHRs, PACS, dictation, or whatever new dumb AI implementation someone is pitching to revolutionize healthcare (but that no one talked to an actual doctor about).

The critical tension is the balance between making features obvious, easy, and possible. You have to have opinions, and you have to choose, because being everything to everyone isn’t an option.

// 01.27.26

Our family had some nice, quiet snow days over the weekend here in Dallas, so I finally finished this historically interesting if morbid Lego mosaic of the first CT scan from 1971 (right frontal GBM):

The Stoics thought everyone should have a memento mori, but the typical skull is so staid.

// 01.26.26

The Unhelpfulness of AI Predictions

01.26.26 // Medicine, Radiology

Most Content is Boring

Most current articles about AI and radiology work don’t help us much with regard to the questions people are constantly asking about predicting the future. Publish or perish is fine, but I’ll admit I also don’t really care how many ways we can say these early tools are sorta maybe helpful sometimes for some people but are also super brittle and unreliable and also the most important thing is that they don’t make things worse, particularly due to bad integration with a legacy workflows. How to incorporate something only borderline useful into a workflow you probably don’t control just isn’t very interesting, and people keep dropping predictions about what things will look like years (or even decades!) in the future based on what we’ve seen so far.

Extrapolating linear growth when we have exponential compute isn’t helpful. But imagining a post-work techno-utopia vs dystopia is also not actionable for anyone. We can skip the hand-wringing and just agree that everything currently available isn’t good enough to change the world but that maybe one day it will be.

We live in a complicated world where many of what we perceive as challenging problems are computationally straightforward (e.g. cumbersome long calculations, playing chess) and other things that can become automatic to most humans are computationally intense (as it turns out, driving a car, though it’s getting better!). Corner cases and uncertainty are, in some sense, the whole game when it comes to automation. Medicine—with its treatment algorithms designed for squishy humans with limited narrative skills and inherently incomplete information—is a combination of both.

Lots of medicine is straightforward and subject to guild protectionism, but some of it is genuinely hard. And sometimes the difference between those two situations is only knowable after the fact. If nothing else, ambiguity is hard, and how much uncertainty and risk to accept has never been codified. For example, many current AI tools are designed for a high negative predictive value and are therefore overly sensitive. That’s fine for a tool to help a human not make a mistake, but that still can lead to cry-wolf problems and certainly wouldn’t work for an autonomous machine.

Balancing uncertainty and service expectations is no joke for humans or machines or the combination.

Even Good Predictions about Important Things Are Challenging to Act On

As I’ve said before, I don’t think the reality of impending change means that we can meaningfully predict the contours of that change or should even try to plan for its eventuality in ways that would be destabilizing in the intervening time. What Hinton got wrong is that predictions about replacement are dangerous because, until replacement actually occurs and/or demand shifts, we need to maintain the status quo. There’s no way to know if any field (whether primary care or even surgery) will be augmented or replaced over an effectively infinite timeline, or if the contours of the work will change and if so to what extent and how fast, and therefore what the correct number of people we need long-term such that we could preemptively adjust the pipeline of training to help us land at the optimal amount of people for every kind of job. It’s not going to happen.

For long training pathways like most physician careers, trying to pretend we can is insanity. Central planning in that fashion almost never works because there are far too many unknowns. What we need to do is continue training people for the world we live in with a plan, desire, and ability to adjust our workforce over time as the world changes. We need less planning and more flexibility. Less preciousness about how we work and more orientation on the desired outcomes.

Until the exact day a computer replaces a human, you need that human. Trying to predict a societal need years in the future is a fool’s errand. We haven’t been able to fill the halls of medicine “correctly” over the past century; why would we be better at doing it now?

The Breast Scenario

Let’s just posit a very narrow scenario to see how futile the situation is. Let’s magically say AI for screening mammography becomes incorporated into major vendors at some point in the next X years, allowing for automated reports, even for callbacks. A computer report and a human report would be indistinguishable, especially since BI-RADS facilitates complete automation of reporting language and organization. (I’m also picking breast imaging for this thought experiment because it’s currently very much in demand, tends to be narrowly subspecialized, and has relatively unique current job offerings.)

Let’s also say that the algorithm performance and regulations are such that, for now, those exams are still read by a human being as well—at least in the short term. Guess what the incremental efficiency gain will be? How many screeners can someone read in that setting? The answer, of course, is that it varies.

Automation bias is going to hit fast, so it’s not clear if someone will be able to read 10-20% more, twice as many, three times as many, or some other multiple. Whatever the number is, the first question becomes: how does that affect breast imaging staffing? Does it change the premium that breast imagers can currently command because of the high volume of high-paying screening exams, particularly those with 3D tomosynthesis? How many nefarious rubber-stampers do we need to change the competitive landscape? Think of how fast turnaround times can be! This of course will happen on the backdrop of continually increasing imaging volumes and technological development.

Okay, then what happens to mammography reimbursement? The RVUs will inevitably change if the effort to do the work changes. How fast will that happen, and how drastic will the change be? The RVU system is a zero-sum game, and undoubtedly, there will be a lot of fighting about any changes to radiology reimbursement that occur thanks to AI tools—but they will occur. Tomosythesis is well-paid and already in crosshairs.

How would the subspecialty handle a shifting balance of non-procedural and procedural work, or remote vs on-site work?

Then, how would the job market respond? Would we see similar jobs with less pay? Or would we see a chilling of the growing lifestyle niche of 4-day workweeks with no evenings or weekends? Would we just be able to better staff growing volumes? When I finished training less than a decade ago, it was common for breast imagers to take general call. It would seem that’s pretty rare now. Medicine, like politics, is local. Of course, breast imaging isn’t operating in a vacuum; this is all taking place within an even larger lifestyle teleradiology trend. But variables don’t play nice and isolate themselves: everything is always changing all the time.

What is the timescale for those changes, and how piecemeal would they be? What are the second and third-order downstream consequences of even a single shift in radiology productivity? What about fully automated DEXA? What about prepopulated ultrasound measurements? Natural language draft reports for other modalities. We don’t even know. But the answer isn’t simply that we need fewer breast imagers when there are currently not enough, just because one day we might.

Medico-legally, how will we as a society handle mistakes made by machines? Or, with a human checker, how will we weigh a change made by a human versus a mistake made by a human coupled with a machine?

How will we feel about a mistake where the AI misses something and the human doesn’t catch it–or doesn’t fix it–versus a situation where the AI calls something and the human pulls it back, even when the AI is right? How will we handle type 1 versus type 2 errors? Will companies self-insure for autonomous work as the dominant strategy, or will we see protectionist legislation about a variety of key human tasks?

We don’t know, long-term, whether there are edge cases where humans are more or less important. But we also don’t know, even short-term, where the contours of those skills will reside. Maybe we’re entering a world where the ED would be a better place if a computer did most of the interviewing and initial workup for stable patients.

We are amidst a jagged frontier where AI is good at some things and bad at others, and we know there is a lag between capacity, implementation, legislation, and regulation. Planning is always a challenge; useful prediction is basically impossible.

So, instead of predicting exactly what will happen, we need to acknowledge the broad potential possibilities that could occur and plan a system to deal with them when/if they do occur, as opposed to trying to nail the blind landing.

We All Just Do Stuff Anyway

I don’t want to call out any behavior as irrational, but we should acknowledge upfront the human tendency to just do whatever we want to do and then create post hoc rationales + explanations + narratives to justify and explain our behavior after the fact. Much of this is unconscious and may be a clever trick of our mushy brain psychology, but that doesn’t make it any less real. Try to be honest with yourself and see why you feel the way you do about just about anything and if the reasons you give yourself actually mesh with reality. Write it down, or you’ll cheat (it’s our nature).

It’s like a medical student saying they like “technology” and “problem solving” and “people coming to the reading room for answers” for why they picked radiology when it was really that it seemed relaxing and quiet with good vibes and plenty of coffee and that people at least respected the radiologists’ intelligence and oh yeah they heard they were paid well.

Over the past decade or so, psychiatry has gone from very uncompetitive to pretty competitive. Is it because mental health is important and less stigmatized? Surely that’s part of it. But maybe also because cash-based private practice in a high-demand field (potentially even done via telehealth) sounds like a nice quality of life and a rare way to practice medicine on your own terms. We always seek single explanations thanks to our bias for narrative, but that doesn’t mean that the compelling storytelling is any more reflective of reality.

The examples of failed central planning and the reliance on narrative are legion.

To reiterate my point: No single person or entity has the answer.

So—

Yes, we need all varieties of doctor now, and we need to keep training all of them until we don’t—if that ever occurs.

Will a job like radiology or even primary care pay the same 10 years from now as it does today, or 20 years from now? That will depend on how much value the human provides within the system, how much work they do, how hard or complex that work is, and the supply/demand balance of the available workforce. Volatility and all kinds of non-AI factors already lead to lots of unpredictability.

There may be golden years of excess income for scalable jobs like radiology as these changes first take place. If senior engineers can use AI to do more projects without hiring as many junior coders, it results in rising incomes for those on top and a rug-pull for those entering the workforce.

Both medicine and society need ongoing, never-ending conversations in order to work on creating adaptable systems. Instead of placing bets, the acronym organizations need to set stakeholders up for success across the full spectrum from an AI-nothingburger to AI-work-is-optional.

Number Games

01.22.26 // Medicine, Miscellany

More from Alchemy: The Dark Art and Curious Science of Creating Magic in Brands, Business, and Life by Rory Sutherland (Part 1 is here):

When every function of a business is looked at from the same narrow economic standpoint, the same game is applied endlessly. Define something narrowly, automate or streamline it – or remove it entirely – then regard the savings as profit. Is this, too, explained by argumentative thinking, where we would rather win an argument than be right?

Especially egregious in large institutions engaging in budgetary arbitrage. A hospital where I work “saved” money a few years ago by outsourcing a lot of its IT abroad. I guarantee they didn’t actually save money for the organization despite spending less on the IT line item.

Today, the principal activity of any publicly held company is rarely the creation of products to satisfy a market need. Management attention is instead largely directed towards the invention of plausible-sounding efficiency narratives to satisfy financial analysts, many of whom know nothing about the businesses they claim to analyse, beyond what they can read on a spreadsheet. There is no need to prove that your cost-saving works empirically, as long as it is consistent with standard economic theory. It is a simple principle of business that, however badly your decision turns out, you will never be fired for following economics, even though its predictive value lies somewhere between water divining and palmistry.

[…]

The problems occur when people try to solve ‘wide’ problems using ‘narrow’ thinking. Keynes once said, ‘It is better to be vaguely right than precisely wrong’, and evolution seems to be on his side. The risk with the growing use of cheap computational power is that it encourages us to take a simple, mathematically expressible part of a complicated question, solve it to a high degree of mathematical precision, and assume we have solved the whole problem.

There’s a line from Tony Fadell’s Build that I love: “Data can’t solve an opinion-based problem.” So many meetings and dashboards, and so much of that data is used as a proxy to somehow “answer” the real question: Is this all really working? Can we do this better? Not just the one part that’s easy to measure—but the whole thing?

In the event, the platform has improved since we adopted it, but the fact that a cost-saving decision could be made without any consideration of the hidden risks to efficiency was nonetheless alarming. Why are large commercial organisations adopting this ideological approach to business? That was supposed to be the weakness of communism.

At least in many healthcare and other large organizations, it’s departmental siloing and abstracting both strategy and tactical decision making to layers of managers with little or out-of-date domain expertise. I have seen even good initial decisions with involved stakeholders ruined by implementation decisions by suits.

People’s motivations are not always well-aligned with the interests of a business: the best decision to make is to pursue rational self-justification, not profit. No one was ever fired for pretending economics was true.

Moral hazard is potent, especially when combined with the Peter Principle.

We fetishise precise numerical answers because they make us look scientific – and we crave the illusion of certainty. But the real genius of humanity lies in being vaguely right – the reason that we do not follow the assumptions of economists about what is rational behaviour is not necessarily because we are stupid. It may be because part of our brain has evolved to ignore the map, or to replace the initial question with another one – not so much to find a right answer as to avoid a disastrously wrong one.

It’s comfortable but wrong to wave away unavoidable uncertainty with a blanket of data.

[Herbert] Simon used satisficing to explain the behaviour of decision makers under circumstances in which an optimal solution cannot be determined. He maintained that many natural problems are characterised by computational intractability or a lack of information, both of which preclude the use of mathematical optimisation procedures.

[…]

Consequently, as he observed in his speech on winning the 1978 Nobel Prize, “decision makers can satisfice either by finding optimum solutions for a simplified world, or by finding satisfactory solutions for a more realistic world.

I’ve included Simon’s work on “bounded rationality” in a bunch of my talks, but surprisingly not much on this site. Simon’s work set the stage for much of behavioral economics and modern decision science. It is of both incredible personal and professional value.

I’m in the new episode 7 of AJR’s Radiology Trailblazers podcast with Dr. Lindsey Negrete and Dr. Amy Maduram. Please do not think for a second that I accept my inclusion in any “trailblazing” premise(!), but we had a nice discussion about the writing process in and out of radiology. Whenever I do a podcast, I am always able to create at least one new verbal tic. This time it was “in reality”—please forgive me.

// 01.21.26

Author Tim O’Reilly, in his 2006 commencement speech at UC Berkeley:

Money is like gasoline during a road trip. You don’t want to run out of gas on your trip, but you’re not doing a tour of gas stations.

// 01.21.26

The Refill Machine

01.16.26 // Medicine

The announcement of ChatGPT Health and then immediately Claude for Healthcare (just for “informational purposes,” of course) is big news, obviously. But the other big news from last week was Doctronic’s new pilot in Utah:

In a first for the U.S., Utah is letting artificial intelligence — not a doctor — renew certain medical prescriptions. No human involved.

The state has launched a pilot program with health-tech startup Doctronic that allows an AI system to handle routine prescription renewals for patients with chronic conditions

The program is limited to 190 commonly prescribed medications, so no pain or ADHD refills will be happening here. Some fighting words from the CEO:

“The AI is actually better than doctors at doing this,” said Dr. Adam Oskowitz, Doctronic co-founder and an associate professor of surgery at the University of California San Francisco. “When you go see a doctor, it’s not going to do all the checks that the AI is doing.”

In medicine, there’s always going to be potential issues that patients have,” said Oskowitz. “Whether it’s caused by the AI or not — we will take the risk. I think this is going to be infinitely safer than a human doctor.”

It’s worth pointing out that it’s much safer and easier for this product to practice medicine in the limited sense of renewing the previous decision of a human than it is to work de novo.

But the access problem is real, and 24/7 telehealth for a variety of urgent care-type problems is going to be a powerful argument in the near future, especially in rural areas.

The tools are improving, but the great irony is that clinical medicine has set the stage for this in two important ways.

1. Medicine is increasingly algorithmic.

The codification of guidelines and best practices means that large swaths of medicine are not just cognitively routine but are supposed to work within narrow variants. Picking from a few defensible (“correct”) options is easier than crafting the tasting menu from scratch.

When we want flesh-and-blood humans to utilize algorithms in the treatment of routine conditions, then a machine can utilize those same algorithms just as well—if not better per the evangelists—especially since a machine currently always has the opportunity to punt to a human if it’s confused. If there are hard stops in place for ethically fraught or problematic prescribing, then the most blatant jailbreaking concerns go down. It’s also worth pointing out that we already have online pill mills these days, no AI needed.

The product is going to punt whenever screener questions suggest medication intolerance or new meds in the reconciliation have interactions. Presumably, anything with nuance or that might require an alternative.

This is simply allowing a computer to refill a prescription that was already provided by a human for a medication taken by a human who still wants more. The algorithm can simply give the refill unless the patient tells it some contraindication has arisen in the intervening months that should change the picture, like a new medication interaction.

2. Healthcare is Terrible and Hard to Get

Another contributing factor is that healthcare is terrible, expensive, and hard to get. Patients wait for a long time and travel far and wide for short appointments that run behind schedule. It’s mostly not the fault of practitioners that they are overly busy, squeezing patients into tiny slots, but that’s the baseline reality. We are increasingly living in an era where access for routine clinical medicine is limited by cost. Those with the financial ability can opt for direct/concierge style care, and the rest are increasingly shuffled into a different tier, increasingly ministered by nonphysician practitioners operating autonomously.

Regardless of the details, it’s easier to replace something that isn’t good.

And let’s be clear with this initial salvo in the doctor-replacement process: this is not an LLM providing comprehensive care—doing an H&P or a new-patient evaluation and deciding how best to treat someone’s hypertension. That is a heavier lift and one prone to a hell of a lot more liability.

This is a machine taking some low-hanging fruit.

Consequences

Harder Average Work

One downstream problem of this approach is that quick med refill visits are some of the ways that a clinic actually makes money, and that facetime can still lead to important, unpredictable, impromtu care. Because so much clinical care is underpaid, you need some easy, straightforward work to keep from falling too far behind in a busy clinic schedule.

Patients disappearing from the clinic rolls, getting refills for a few years, and then coming back when they have a problem will be a challenge.

A possible early hole in the “Human in the Loop” argument

You could argue a human should be in the loop. But I think the reality of automation bias is that this is a difficult argument. A human being reviewing tons and tons of these types of AI decisions probably isn’t actually going to pay enough attention to catch errors at a meaningful frequency. One could reasonably debate how often these mistakes are currently caught during 15-minute med checks. But in many cases, in many systems, many, many refill requests populate in the EHR and just require a button press anyway.

A patient doesn’t always have to come in for an appointment just to check a box and get a refill at baseline, but on the whole a pilot like this will result in fewer upfront visits, shorten turnaround time, save time, money, and physician computer clicks. Some of the negative consequences will be anecdotes, and the real second-order effects will probably be missed. By taking on the minimal liability for doing this work, the AI company can charge for what amounts to MyChart messages.

Liability Isn’t the Magic Moat

People often talk about liability as if only human beings, doing the work of humans, could ever be insured. But that is obviously, manifestly untrue. Many of our current consumer insurance products (e.g. home insurance) are designed to deal with a variety of bad outcomes, not just bad human causes.

If an insurance company does the math and wants to underwrite a venture, it can. If a large company is willing to pay its own damages, it can self-insure.

What makes something worth doing is a risk-benefit ratio where, if the profits are high enough to counteract the risks and still generate a profit, it’s something pursuable in our current system.

If regulators don’t mind, a company or even a health system could choose to implement any variety of these products and self-insure, even without an overhaul of our whole medical-legal system.

In the End

There is no doubt that some parts of telehealth are, frankly, inadequate for care today. We don’t really get to the next level of the AI version without a robust multimodal approach that actually incorporates computer vision to look at a patient and see how they’re doing, auditory analysis for tone and voice changes, and real-time conversational language analysis.

This product can skip all that. It’s not providing human-level care. It’s acknowledging the reality that a lot of medicine is routine with a low bar.

In radiology, for example, people have previously argued that we would have machines read the normal cases autonomously and involve human radiologists just to do the ones that are abnormal to check. If this were plausible, it would mean that the average complexity per case goes up, and the actual work per case gets harder.

It may make sense in some cases, but operating at the top of your license, so to speak, can also be exhausting.

It remains to be seen if there is a feasible way for any field of practice to do this in a sustainable way, even if it might be an economically viable one.