Basecamp’s Jason Fried’s brief discussion of product development should be required reading for anyone who designs software, including EHRs, PACS, dictation, or whatever new dumb AI implementation someone is pitching to revolutionize healthcare (but that no one talked to an actual doctor about).
The critical tension is the balance between making features obvious, easy, and possible. You have to have opinions, and you have to choose, because being everything to everyone isn’t an option.
Our family had some nice, quiet snow days over the weekend here in Dallas, so I finally finished this historically interesting if morbid Lego mosaic of the first CT scan from 1971 (right frontal GBM):
The Stoics thought everyone should have a memento mori, but the typical skull is so staid.
Most current articles about AI and radiology work don’t help us much with regard to the questions people are constantly asking about predicting the future. Publish or perish is fine, but I’ll admit I also don’t really care how many ways we can say these early tools are sorta maybe helpful sometimes for some people but are also super brittle and unreliable and also the most important thing is that they don’t make things worse, particularly due to bad integration with a legacy workflows. How to incorporate something only borderline useful into a workflow you probably don’t control just isn’t very interesting, and people keep dropping predictions about what things will look like years (or even decades!) in the future based on what we’ve seen so far.
Extrapolating linear growth when we have exponential compute isn’t helpful. But imagining a post-work techno-utopia vs dystopia is also not actionable for anyone. We can skip the hand-wringing and just agree that everything currently available isn’t good enough to change the world but that maybe one day it will be.
We live in a complicated world where many of what we perceive as challenging problems are computationally straightforward (e.g. cumbersome long calculations, playing chess) and other things that can become automatic to most humans are computationally intense (as it turns out, driving a car, though it’s getting better!). Corner cases and uncertainty are, in some sense, the whole game when it comes to automation. Medicine—with its treatment algorithms designed for squishy humans with limited narrative skills and inherently incomplete information—is a combination of both.
Lots of medicine is straightforward and subject to guild protectionism, but some of it is genuinely hard. And sometimes the difference between those two situations is only knowable after the fact. If nothing else, ambiguity is hard, and how much uncertainty and risk to accept has never been codified. For example, many current AI tools are designed for a high negative predictive value and are therefore overly sensitive. That’s fine for a tool to help a human not make a mistake, but that still can lead to cry-wolf problems and certainly wouldn’t work for an autonomous machine.
Balancing uncertainty and service expectations is no joke for humans or machines or the combination.
As I’ve said before, I don’t think the reality of impending change means that we can meaningfully predict the contours of that change or should even try to plan for its eventuality in ways that would be destabilizing in the intervening time. What Hinton got wrong is that predictions about replacement are dangerous because, until replacement actually occurs and/or demand shifts, we need to maintain the status quo. There’s no way to know if any field (whether primary care or even surgery) will be augmented or replaced over an effectively infinite timeline, or if the contours of the work will change and if so to what extent and how fast, and therefore what the correct number of people we need long-term such that we could preemptively adjust the pipeline of training to help us land at the optimal amount of people for every kind of job. It’s not going to happen.
For long training pathways like most physician careers, trying to pretend we can is insanity. Central planning in that fashion almost never works because there are far too many unknowns. What we need to do is continue training people for the world we live in with a plan, desire, and ability to adjust our workforce over time as the world changes. We need less planning and more flexibility. Less preciousness about how we work and more orientation on the desired outcomes.
Until the exact day a computer replaces a human, you need that human. Trying to predict a societal need years in the future is a fool’s errand. We haven’t been able to fill the halls of medicine “correctly” over the past century; why would we be better at doing it now?
Let’s just posit a very narrow scenario to see how futile the situation is. Let’s magically say AI for screening mammography becomes incorporated into major vendors at some point in the next X years, allowing for automated reports, even for callbacks. A computer report and a human report would be indistinguishable, especially since BI-RADS facilitates complete automation of reporting language and organization. (I’m also picking breast imaging for this thought experiment because it’s currently very much in demand, tends to be narrowly subspecialized, and has relatively unique current job offerings.)
Let’s also say that the algorithm performance and regulations are such that, for now, those exams are still read by a human being as well—at least in the short term. Guess what the incremental efficiency gain will be? How many screeners can someone read in that setting? The answer, of course, is that it varies.
Automation bias is going to hit fast, so it’s not clear if someone will be able to read 10-20% more, twice as many, three times as many, or some other multiple. Whatever the number is, the first question becomes: how does that affect breast imaging staffing? Does it change the premium that breast imagers can currently command because of the high volume of high-paying screening exams, particularly those with 3D tomosynthesis? How many nefarious rubber-stampers do we need to change the competitive landscape? Think of how fast turnaround times can be! This of course will happen on the backdrop of continually increasing imaging volumes and technological development.
Okay, then what happens to mammography reimbursement? The RVUs will inevitably change if the effort to do the work changes. How fast will that happen, and how drastic will the change be? The RVU system is a zero-sum game, and undoubtedly, there will be a lot of fighting about any changes to radiology reimbursement that occur thanks to AI tools—but they will occur. Tomosythesis is well-paid and already in crosshairs.
How would the subspecialty handle a shifting balance of non-procedural and procedural work, or remote vs on-site work?
Then, how would the job market respond? Would we see similar jobs with less pay? Or would we see a chilling of the growing lifestyle niche of 4-day workweeks with no evenings or weekends? Would we just be able to better staff growing volumes? When I finished training less than a decade ago, it was common for breast imagers to take general call. It would seem that’s pretty rare now. Medicine, like politics, is local. Of course, breast imaging isn’t operating in a vacuum; this is all taking place within an even larger lifestyle teleradiology trend. But variables don’t play nice and isolate themselves: everything is always changing all the time.
What is the timescale for those changes, and how piecemeal would they be? What are the second and third-order downstream consequences of even a single shift in radiology productivity? What about fully automated DEXA? What about prepopulated ultrasound measurements? Natural language draft reports for other modalities. We don’t even know. But the answer isn’t simply that we need fewer breast imagers when there are currently not enough, just because one day we might.
Medico-legally, how will we as a society handle mistakes made by machines? Or, with a human checker, how will we weigh a change made by a human versus a mistake made by a human coupled with a machine?
How will we feel about a mistake where the AI misses something and the human doesn’t catch it–or doesn’t fix it–versus a situation where the AI calls something and the human pulls it back, even when the AI is right? How will we handle type 1 versus type 2 errors? Will companies self-insure for autonomous work as the dominant strategy, or will we see protectionist legislation about a variety of key human tasks?
We don’t know, long-term, whether there are edge cases where humans are more or less important. But we also don’t know, even short-term, where the contours of those skills will reside. Maybe we’re entering a world where the ED would be a better place if a computer did most of the interviewing and initial workup for stable patients.
We are amidst a jagged frontier where AI is good at some things and bad at others, and we know there is a lag between capacity, implementation, legislation, and regulation. Planning is always a challenge; useful prediction is basically impossible.
So, instead of predicting exactly what will happen, we need to acknowledge the broad potential possibilities that could occur and plan a system to deal with them when/if they do occur, as opposed to trying to nail the blind landing.
I don’t want to call out any behavior as irrational, but we should acknowledge upfront the human tendency to just do whatever we want to do and then create post hoc rationales + explanations + narratives to justify and explain our behavior after the fact. Much of this is unconscious and may be a clever trick of our mushy brain psychology, but that doesn’t make it any less real. Try to be honest with yourself and see why you feel the way you do about just about anything and if the reasons you give yourself actually mesh with reality. Write it down, or you’ll cheat (it’s our nature).
It’s like a medical student saying they like “technology” and “problem solving” and “people coming to the reading room for answers” for why they picked radiology when it was really that it seemed relaxing and quiet with good vibes and plenty of coffee and that people at least respected the radiologists’ intelligence and oh yeah they heard they were paid well.
Over the past decade or so, psychiatry has gone from very uncompetitive to pretty competitive. Is it because mental health is important and less stigmatized? Surely that’s part of it. But maybe also because cash-based private practice in a high-demand field (potentially even done via telehealth) sounds like a nice quality of life and a rare way to practice medicine on your own terms. We always seek single explanations thanks to our bias for narrative, but that doesn’t mean that the compelling storytelling is any more reflective of reality.
The examples of failed central planning and the reliance on narrative are legion.
To reiterate my point: No single person or entity has the answer.
Yes, we need all varieties of doctor now, and we need to keep training all of them until we don’t—if that ever occurs.
Will a job like radiology or even primary care pay the same 10 years from now as it does today, or 20 years from now? That will depend on how much value the human provides within the system, how much work they do, how hard or complex that work is, and the supply/demand balance of the available workforce. Volatility and all kinds of non-AI factors already lead to lots of unpredictability.
There may be golden years of excess income for scalable jobs like radiology as these changes first take place. If senior engineers can use AI to do more projects without hiring as many junior coders, it results in rising incomes for those on top and a rug-pull for those entering the workforce.
Both medicine and society need ongoing, never-ending conversations in order to work on creating adaptable systems. Instead of placing bets, the acronym organizations need to set stakeholders up for success across the full spectrum from an AI-nothingburger to AI-work-is-optional.
More from Alchemy: The Dark Art and Curious Science of Creating Magic in Brands, Business, and Life by Rory Sutherland (Part 1 is here):
When every function of a business is looked at from the same narrow economic standpoint, the same game is applied endlessly. Define something narrowly, automate or streamline it – or remove it entirely – then regard the savings as profit. Is this, too, explained by argumentative thinking, where we would rather win an argument than be right?
Especially egregious in large institutions engaging in budgetary arbitrage. A hospital where I work “saved” money a few years ago by outsourcing a lot of its IT abroad. I guarantee they didn’t actually save money for the organization despite spending less on the IT line item.
Today, the principal activity of any publicly held company is rarely the creation of products to satisfy a market need. Management attention is instead largely directed towards the invention of plausible-sounding efficiency narratives to satisfy financial analysts, many of whom know nothing about the businesses they claim to analyse, beyond what they can read on a spreadsheet. There is no need to prove that your cost-saving works empirically, as long as it is consistent with standard economic theory. It is a simple principle of business that, however badly your decision turns out, you will never be fired for following economics, even though its predictive value lies somewhere between water divining and palmistry.
[…]
The problems occur when people try to solve ‘wide’ problems using ‘narrow’ thinking. Keynes once said, ‘It is better to be vaguely right than precisely wrong’, and evolution seems to be on his side. The risk with the growing use of cheap computational power is that it encourages us to take a simple, mathematically expressible part of a complicated question, solve it to a high degree of mathematical precision, and assume we have solved the whole problem.
There’s a line from Tony Fadell’s Build that I love: “Data can’t solve an opinion-based problem.” So many meetings and dashboards, and so much of that data is used as a proxy to somehow “answer” the real question: Is this all really working? Can we do this better? Not just the one part that’s easy to measure—but the whole thing?
In the event, the platform has improved since we adopted it, but the fact that a cost-saving decision could be made without any consideration of the hidden risks to efficiency was nonetheless alarming. Why are large commercial organisations adopting this ideological approach to business? That was supposed to be the weakness of communism.
At least in many healthcare and other large organizations, it’s departmental siloing and abstracting both strategy and tactical decision making to layers of managers with little or out-of-date domain expertise. I have seen even good initial decisions with involved stakeholders ruined by implementation decisions by suits.
People’s motivations are not always well-aligned with the interests of a business: the best decision to make is to pursue rational self-justification, not profit. No one was ever fired for pretending economics was true.
Moral hazard is potent, especially when combined with the Peter Principle.
We fetishise precise numerical answers because they make us look scientific – and we crave the illusion of certainty. But the real genius of humanity lies in being vaguely right – the reason that we do not follow the assumptions of economists about what is rational behaviour is not necessarily because we are stupid. It may be because part of our brain has evolved to ignore the map, or to replace the initial question with another one – not so much to find a right answer as to avoid a disastrously wrong one.
It’s comfortable but wrong to wave away unavoidable uncertainty with a blanket of data.
[Herbert] Simon used satisficing to explain the behaviour of decision makers under circumstances in which an optimal solution cannot be determined. He maintained that many natural problems are characterised by computational intractability or a lack of information, both of which preclude the use of mathematical optimisation procedures.
[…]
Consequently, as he observed in his speech on winning the 1978 Nobel Prize, “decision makers can satisfice either by finding optimum solutions for a simplified world, or by finding satisfactory solutions for a more realistic world.
I’ve included Simon’s work on “bounded rationality” in a bunch of my talks, but surprisingly not much on this site. Simon’s work set the stage for much of behavioral economics and modern decision science. It is of both incredible personal and professional value.
I’m in the new episode 7 of AJR’s Radiology Trailblazers podcast with Dr. Lindsey Negrete and Dr. Amy Maduram. Please do not think for a second that I accept my inclusion in any “trailblazing” premise(!), but we had a nice discussion about the writing process in and out of radiology. Whenever I do a podcast, I am always able to create at least one new verbal tic. This time it was “in reality”—please forgive me.
Author Tim O’Reilly, in his 2006 commencement speech at UC Berkeley:
Money is like gasoline during a road trip. You don’t want to run out of gas on your trip, but you’re not doing a tour of gas stations.
The announcement of ChatGPT Health and then immediately Claude for Healthcare (just for “informational purposes,” of course) is big news, obviously. But the other big news from last week was Doctronic’s new pilot in Utah:
In a first for the U.S., Utah is letting artificial intelligence — not a doctor — renew certain medical prescriptions. No human involved.
The state has launched a pilot program with health-tech startup Doctronic that allows an AI system to handle routine prescription renewals for patients with chronic conditions
The program is limited to 190 commonly prescribed medications, so no pain or ADHD refills will be happening here. Some fighting words from the CEO:
“The AI is actually better than doctors at doing this,” said Dr. Adam Oskowitz, Doctronic co-founder and an associate professor of surgery at the University of California San Francisco. “When you go see a doctor, it’s not going to do all the checks that the AI is doing.”
In medicine, there’s always going to be potential issues that patients have,” said Oskowitz. “Whether it’s caused by the AI or not — we will take the risk. I think this is going to be infinitely safer than a human doctor.”
It’s worth pointing out that it’s much safer and easier for this product to practice medicine in the limited sense of renewing the previous decision of a human than it is to work de novo.
But the access problem is real, and 24/7 telehealth for a variety of urgent care-type problems is going to be a powerful argument in the near future, especially in rural areas.
The tools are improving, but the great irony is that clinical medicine has set the stage for this in two important ways.
The codification of guidelines and best practices means that large swaths of medicine are not just cognitively routine but are supposed to work within narrow variants. Picking from a few defensible (“correct”) options is easier than crafting the tasting menu from scratch.
When we want flesh-and-blood humans to utilize algorithms in the treatment of routine conditions, then a machine can utilize those same algorithms just as well—if not better per the evangelists—especially since a machine currently always has the opportunity to punt to a human if it’s confused. If there are hard stops in place for ethically fraught or problematic prescribing, then the most blatant jailbreaking concerns go down. It’s also worth pointing out that we already have online pill mills these days, no AI needed.
The product is going to punt whenever screener questions suggest medication intolerance or new meds in the reconciliation have interactions. Presumably, anything with nuance or that might require an alternative.
This is simply allowing a computer to refill a prescription that was already provided by a human for a medication taken by a human who still wants more. The algorithm can simply give the refill unless the patient tells it some contraindication has arisen in the intervening months that should change the picture, like a new medication interaction.
Another contributing factor is that healthcare is terrible, expensive, and hard to get. Patients wait for a long time and travel far and wide for short appointments that run behind schedule. It’s mostly not the fault of practitioners that they are overly busy, squeezing patients into tiny slots, but that’s the baseline reality. We are increasingly living in an era where access for routine clinical medicine is limited by cost. Those with the financial ability can opt for direct/concierge style care, and the rest are increasingly shuffled into a different tier, increasingly ministered by nonphysician practitioners operating autonomously.
Regardless of the details, it’s easier to replace something that isn’t good.
And let’s be clear with this initial salvo in the doctor-replacement process: this is not an LLM providing comprehensive care—doing an H&P or a new-patient evaluation and deciding how best to treat someone’s hypertension. That is a heavier lift and one prone to a hell of a lot more liability.
This is a machine taking some low-hanging fruit.
One downstream problem of this approach is that quick med refill visits are some of the ways that a clinic actually makes money, and that facetime can still lead to important, unpredictable, impromtu care. Because so much clinical care is underpaid, you need some easy, straightforward work to keep from falling too far behind in a busy clinic schedule.
Patients disappearing from the clinic rolls, getting refills for a few years, and then coming back when they have a problem will be a challenge.
You could argue a human should be in the loop. But I think the reality of automation bias is that this is a difficult argument. A human being reviewing tons and tons of these types of AI decisions probably isn’t actually going to pay enough attention to catch errors at a meaningful frequency. One could reasonably debate how often these mistakes are currently caught during 15-minute med checks. But in many cases, in many systems, many, many refill requests populate in the EHR and just require a button press anyway.
A patient doesn’t always have to come in for an appointment just to check a box and get a refill at baseline, but on the whole a pilot like this will result in fewer upfront visits, shorten turnaround time, save time, money, and physician computer clicks. Some of the negative consequences will be anecdotes, and the real second-order effects will probably be missed. By taking on the minimal liability for doing this work, the AI company can charge for what amounts to MyChart messages.
People often talk about liability as if only human beings, doing the work of humans, could ever be insured. But that is obviously, manifestly untrue. Many of our current consumer insurance products (e.g. home insurance) are designed to deal with a variety of bad outcomes, not just bad human causes.
If an insurance company does the math and wants to underwrite a venture, it can. If a large company is willing to pay its own damages, it can self-insure.
What makes something worth doing is a risk-benefit ratio where, if the profits are high enough to counteract the risks and still generate a return, it’s something pursuable in our current system.
If regulators don’t mind, a company or even a health system could choose to implement any variety of these products and self-insure, even without an overhaul of our whole medical-legal system.
There is no doubt that some parts of telehealth are, frankly, inadequate for care today. We also won’t really get to the next level of the AI version without a robust multimodal approach that actually incorporates computer vision to look at a patient and see how they’re doing, auditory analysis for tone and voice changes, and real-time conversational language analysis.
This product can skip all that. It’s not providing human-level care. It’s acknowledging the reality that a lot of medicine is routine with a low bar.
In radiology, for example, people have previously argued that we would have machines read the normal cases autonomously and involve human radiologists to check the abnormal ones. If this were plausible, it would mean that the average complexity per case goes up, and the actual work per case gets harder.
It may make sense in some cases, but operating at the top of your license, so to speak, can also be exhausting.
It remains to be seen if there is a feasible way for any field of practice to do this in a sustainable way, even if it might be an economically viable one.
From Developmental Editing by Scott Norton:
Few pleasures are as great as the taste of a fresh idea. A new insight melts in the brain like chocolate on the tongue. Whether the insight is unprecedented in human history or news only to yourself doesn’t matter; the first time a thought occurs is always magic.
That magic is so fickle, perishable. I always find the strong desire to capture as much of it as possible, and the more I can horde upfront the better chance I have of making it to the finish line of anything.
A paraphrased reader question:
I want to be a daytime tele neurorad. Why do so many of the listings seem to be for body? Are there too many neurorads?
I don’t think it’s really that there are too many neuroradiologists per se. It’s that the true need across the widest variety of practice types is general radiology.
If body imagers only read specialized body MRI, they wouldn’t be filling the holes that we have as a field. The greatest need is for plain films, ultrasound, and generic CT.
That is generally part of the job for body imagers, but many neurorads (and other subspecialties as well) really want to read more within their subspecialty, leaving a pile of general radiology for which groups are desperate enough to bring on remote readers.
In reality, there is nothing about a thyroid ultrasound that should make it a “body” examination and not a neuro one, given that the thyroid gland is in the neck. But this is the way everyone practices.
As a result, if a group hires more neurorads to support general needs without needing a full neuro FTE, that dilutes the available neuro work and requires the neurorads to start reading more general radiology. They, generally, don’t want to do that. And in practices where neurorads are 100% or nearly fully subspecialized, they don’t have the ability to overflow into another subspecialty easily.
Staffing in body typically doesn’t really work that way: smaller changes to their casemix aren’t going to result in a super different job or fundamentally change the spectrum of cases they interpret. No new skills needed.
And therefore, body is the greatest need. Not because there are an infinite number of liver MRIs to read (though there are a lot), but rather because we need people who are willing to do basic radiology work. And many body radiologists are expected to do that.
At least that’s my impression of the current state.
There is daytime tele neuro work out there, but I agree it’s mostly if you’re also willing to also read general too.
From Michael Porter’s Competitive Strategy:
We usually think of suppliers as other firms, but labor must be recognized as a supplier as well, and one that exerts great power in many industries. There is substantial empirical evidence that scarce, highly skilled employees and/or tightly unionized labor can bargain away a significant fraction of potential profits in an industry.
This is one of the core tensions in modern healthcare. Healthcare workers are the key source of revenue, but—because they want to be paid pesky competitive wages—also erode the potential profits of their employers, and many of those companies (regardless of their technical nonprofit vs profit status) really, really want more profit.
Private practice is not a panacea, but as most healthcare workers are not unionized, plausible alternatives to large entity employment are critical to enable workers to bargain for fair compensation. I suspect this is an increasing tension we should continue to expect with the greater use of AI. Employed physicians should desperately want their private practice colleagues to succeed.
From Porter’s follow-up, Redefining Health Care: Creating Value-based Competition on Results:
The fundamental problem in the U.S. health care system is that the structure of health care delivery is broken. This is what all the data about rising costs and alarming quality are telling us. And the structure of health care delivery is broken because competition is broken. All of the well-intended reform movements have failed because they did not address the underlying nature of competition.
Things have gotten so, so much worse since that was published in 2006.