I recently experienced a trial of Tesla’s full self-driving capability for the first time. It was a decent, if bewildering and somewhat spooky experience. It’s unusual to see a computer do a task in the real world that you have been doing—and had to do yourself—for decades.

For some reason, feeling a computer manifesting its judgment and controlling a vehicle in three-dimensional space does feel a little bit different than seeing its capabilities in a chat box. In some ways, it is less impressive, and in others, much more.

Certainly, the current iteration—which requires you to maintain attention on the road—does not fulfill the promise of true self-driving in the sense that you are still, at least in part, the driver of the car and ultimately responsible for it, as opposed to being a front-seat passenger able to do other tasks. And that’s for good reason.

The reality is that, like with human drivers, the issues are always in the edge cases. The average person does not get in a crash after every left turn, and neither do self-driving cars. I was struck by how easy it is to trust, and how willing I was to let the car give it a go. The Tesla did struggle repeatedly in my experience in several locations. It may drive more miles without an accident than the average human driver, but the mistakes it makes currently are also ones that most people would not make:

It gets caught in turn-only lanes or the wrong turning lanes pretty consistently and doesn’t learn from those mistakes. It doesn’t, for example, move over nearly fast enough for a right-hand turn after exiting the highway if there is a long line of cars waiting. More dramatically, in downtown Dallas’s web of one-way streets, it tried to go down the wrong way in a bifurcation into oncoming traffic. It also ran a red light after trailing an 18-wheeler so close that it couldn’t see the traffic light.

That being said, what struck me most is how quickly humans trust—and how rapidly automation bias takes effect simply by virtue of the product being available. There is a serious sense of trust in its capabilities. You see it handle a couple of turns, stop appropriately, go appropriately, and check the blind spot before changing lanes, and suddenly, you believe it can do the real deal. You’re ready to trust it on the road in ways that should frankly be pretty surprising. You begin to resent its incessant demand that you pay attention.

In some post-AGI arguments, some people say that humans like other humans and really want other humans in the loop—but I’m actually not sure that’s the case. Especially not if the lack of humans in the loop means faster service, cheaper service, or more customizable service. Really, every situation could be different, and how you feel about your doctor or your pastor might be different than how you feel about yourself as a car or about cabbies.

But these sorts of shortcomings are probably a matter of time, and the downstream consequences of even just this narrow use are profound and unknowable. I don’t want to opine on American car culture or how cities might deal with fleets of self-driving cars waiting for passengers.

It does make you wonder how high the bar for AI adoption is for any particular task. If we can trust a car so easily, a matter of personal life or death, how fast will the shelf life of human expertise decay in other contexts?

In healthcare, for example, I have seen nothing autonomous that is sufficiently reliable yet, which is why adoption outside of personal use or low-hanging general LLM fruit—scribing, summaries/impressions, some clinical decision support/brainstorming, etc.—is relatively low other than narrow models flagging stuff like pulmonary emboli or evaluating a single finding like diabetic retinopathy or bone age. If, with broader tool adoption, we still need a human with hands on the wheel to prevent catastrophe in rare occurrences, then we will need to set up a system carefully to keep that human paying attention and not just going through the motions of looking like they are paying attention in order to placate a monitoring algorithm instead of actually focusing on the road.

The question, as always, becomes: is the purpose to be more efficient, or is the purpose to provide a better product? Those are, as always, in constant tension.

If you search for reviews or commentary on Tesla self-driving, the most common thing you’ll find is frustration that the car won’t let you stop paying attention. Autopilot is frustrating. Driving is fine. Being a passenger is fine. Being in between (other than in stop-and-go traffic) is kinda annoying.

This is likely a temporary problem, and while driving is also different from many people’s jobs, I think it does speak to an underlying human desire: people do not like feeling superfluous. I do not think as a species we will like being forced to rubber-stamp or clock in just for the pretense of being a human in the loop (i.e. drawing a paycheck as liability operator/sin-eater).

Real work is meaningful. Box-checking is soul-sucking, demoralizing, and breeds resentment.

Karl Ove Knausgaard, writing in Harper’s earlier this year:

It feels as if the whole world has been transformed into images of the world and has thus been drawn into the human realm, which now encompasses everything. There is no place, no thing, no person or phenomenon that I cannot obtain as image or information. One might think this adds substance to the world, since one knows more about it, not less, but the opposite is true: it empties the world, it becomes thinner. That’s because knowledge of the world and experience of the world are two fundamentally different things. While knowledge has no particular time or place and can be transmitted, experience is tied to a specific time and place and can never be repeated. For the same reason, it also can’t be predicted. Exactly those two dimensions – the unrepeatable and the unpredictable – are what technology abolishes. The feeling is one of loss of the world.

Knausgaard is best known for his voluminous autobiographical writing (his memoirs are in six[!] volumes). It’s perhaps not surprising then that this piece is less about the and more about his experience of technology. I don’t know about the prediction limitation (Asimov might disagree), but I still think he’s on to something.

One corollary to education is that while simulation is powerful and even necessary, the emotional valence of true experience is impossible to replicate entirely.

All the images I’ve seen of places I’ve never been, people I’ve never met create a kind of pseudomemory from a pseudoworld that I don’t participate in.

We are physical creatures living in a physical world with other physical creatures. The question is—as was perhaps most enjoyably explored in the nostalgia romp that is Ready Player One—how much is lost in the virtual world, how strong is the draw to the physical world, and how much of it do we need to live full lives?

Popular essays about AI published in our current media like to cycle between utopianism to massive dystopian automation/disruption to the “plea for collaboration.” The latter, from “A Better Way to Think About AI” by David Autor and James Manyika in The Atlantic:

In any given application, AI is going to automate or it’s going to collaborate, depending on how we design it and how someone chooses to use it. And the distinction matters because bad automation tools—machines that attempt but fail to fully automate a task—also make bad collaboration tools. They don’t merely fall short of their promise to replace human expertise at higher performance or lower cost, they interfere with human expertise, and sometimes undermine it.

This is the no man’s land that explains why articles about AI in medicine don’t reflect reality and why everyone I talk to thinks that all radiologists spend their days awash in useful AI.

Human expertise has a limited shelf life. When machines provide automation, human attention wanders and capabilities decay. This poses no problem if the automation works flawlessly or if its failure (perhaps due to something as mundane as a power outage) doesn’t create a real-time emergency requiring human intervention. But if human experts are the last fail-safe against catastrophic failure of an automated system—as is currently true in aviation—then we need to vigilantly ensure that humans attain and maintain expertise.

The permanent cousin of automation bias will be de-skilling. Pilots who can’t actually take the yoke and land planes anymore are de-skilled. If there is a gap between useful AI and magical super-human AI, then mitigating de-skilling and preventing never-skilling are critical components to any future workflow:

Research on people’s use of AI makes the downsides of this automation mindset ever more apparent. For example, while experts use chatbots as collaboration tools—riffing on ideas, clarifying intuitions—novices often treat them mistakenly as automation tools, oracles that speak from a bottomless well of knowledge. That becomes a problem when an AI chatbot confidently provides information that is misleading, speculative, or simply false. Because current AIs don’t understand what they don’t understand, those lacking the expertise to identify flawed reasoning and outright errors may be led astray.

The seduction of cognitive automation helps explain a worrying pattern: AI tools can boost the productivity of experts but may also actively mislead novices in expertise-heavy fields such as legal services. Novices struggle to spot inaccuracies and lack efficient methods for validating AI outputs. And methodically fact-checking every AI suggestion can negate any time savings.

Beyond the risk of errors, there is some early evidence that overreliance on AI can impede the development of critical thinking, or inhibit learning. Studies suggest a negative correlation between frequent AI use and critical-thinking skills, likely due to increased “cognitive offloading”—letting the AI do the thinking. In high-stakes environments, this tendency toward overreliance is particularly dangerous: Users may accept incorrect AI suggestions, especially if delivered with apparent confidence.

The rise of highly capable assistive AI tools also risks disrupting traditional pathways for expertise development when it’s still clearly needed now, and will be in the foreseeable future. When AI systems can perform tasks previously assigned to research assistants, surgical residents, and pilots, the opportunities for apprenticeship and learning-by-doing disappear. This threatens the future talent pipeline, as most occupations rely on experiential learning—like those radiology residents discussed above.

Learners will not truly learn if they don’t take on tasks independently.