There’s something we’ve started doing, almost without realizing it. Before we know what we think about something, we ask an interface. There’s nothing wrong with asking. What I want to look at here is something else: what happens to us when that question starts to replace the process of thinking for ourselves, instead of accompanying it. Because the difference between the two doesn’t show up in the result. It shows up inside us, in a layer we almost never notice.
Language Models and Learning
I put together a selection of studies that conclude, at first glance, that the use of Artificial Intelligence improves learning:
- A review of sixty-nine experimental studies found that ChatGPT improves academic performance and a propensity toward higher-order thinking.
- A subsequent meta-analysis, covering thirty-five studies and more than four thousand participants, measured a moderate positive effect.
On the level of measurable output, using Language Models (LLMs) as a cognitive prosthesis works. But the first of these two papers, the one reporting the improvement, also reports a detail we need to put on the table: using Language Models “reduces mental effort.”
I think the cleanest experiment for examining that contrast is the following one: a team took one hundred and seventeen university students and split them into four groups (assisted by ChatGPT, by an expert human tutor, by a checklist, and, finally, a control group with no assistance at all). The ChatGPT group wrote the “best” essays. Not only did it beat the control group: it beat the group assisted by a human expert. But—and here’s the crucial part—”there was no knowledge transfer or increase in intrinsic motivation.” The authors name the phenomenon plainly, calling it “metacognitive laziness”: the group that used Language Models to produce their essay got a better text in the eyes of the graders, but its members ended up learning less about the subject.
There’s nothing wrong with asking. What I want to look at here is something else: what happens to us when that question starts to replace the process of thinking for ourselves, instead of accompanying it.
The physiological measurements point in the same direction. In a pre-print study from the MIT Media Lab, fifty-four students wrote essays with their brain activity monitored. “The group that used Language Models showed the weakest connectivity of the three, their texts grew increasingly similar to one another, and the cognitive debt persisted when they were later asked to write without assistance.” The more assistance, the worse the learning. At the population level, another study of six hundred and sixty-six people found a negative correlation between frequent use of AI tools and critical thinking, mediated by cognitive offloading.
I don’t want to bore you any further with empirical data, so let me move on to boring you with my own reflections: I think there’s a pattern here. The cognitive prosthesis works, and that’s precisely why it’s dangerous: “it works so well at the level of the result that it masks the atrophy happening at the level of the process.” And that atrophy is exactly what, over time, leaves us cognitively useless without the prosthesis. A rapid improvement in certain benchmarks, in exchange for cognitive dependence when it comes to representing and evolving those very benchmarks.
I also think it’s important to stress that almost all of this evidence concerns university populations—”adults with a more settled, but still ongoing, cognitive development.” In elementary and secondary school, sample sizes are small and studies are scarce. We’re pushing the use of a prosthesis without yet knowing what its impact might be on still-developing cognition.
The sovereign decision
Juan Ruocco has been talking for a while now about cognitive sovereignty as the capacity to keep one’s own act of thinking active, to filter what one receives, and to resist manipulation. If learning is delegated entirely, it’s not just a skill that gets lost. Degrees of sovereignty get lost too. “And whoever has less sovereignty over their own thinking has less room to imagine, decide, and represent the world in a way that is their own, or agreed upon with others.”
A few months ago I ran an experiment in a Design class. There were about six groups, each with five members, and we had little time to work, so the basic idea was this: given a concrete brief (“Help people who can’t sleep fall asleep”), they had 20 minutes to develop a solution using Generative AI, and, separately, 20 minutes to think through how they’d approach the problem with no phone or computer. At presentation time, the students were unanimously thrilled with the gorgeous landing pages, complete with award-worthy animations, that Claude or Figma had put together for them, and fairly disappointed with their roadmap of 15 or 20 ideas connected on paper.
The strange thing, to close this out, is that none of them were aware of the decisions made in the first case, and they couldn’t grasp the richness of having made experimental decisions on that piece of paper which looked like a poor result but actually held something crucial: they had decided it, and agreed on it together. What they held in their hands wasn’t a bad result. It was the record of a whole set of decisions: why this and not that, what they discarded, what they couldn’t agree on. The same group also had a beautiful landing page, and no understanding whatsoever of why it looked the way it did. They had received answers, not questions.
Hyperreality and the ex-designer
There’s a twist I want to add here, because I find it the most uncomfortable one. Baudrillard spent his career showing how signs detach from the real until they no longer refer to anything but other signs: copies without an original, what he called hyperreality. Language Models are the literal version of that. They don’t copy an original. They generate an average of everything they digested, patterns of patterns, echoes of echoes in an empty cathedral. Something that seems new and is pure iteration.
The problem for cognitive sovereignty runs deeper than it seems. The classic idea of sovereignty assumed there was an authentic “you” behind the manipulation, something worth defending. What if that “you” is also starting to look like the average?
We’re becoming, slowly but not that slowly, ex-designers and ex-programmers. People who choose from a menu built by someone else. And choosing from someone else’s menu isn’t agency: it’s the most comfortable form of obedience.
You start to see it in everyday production settings: for instance, the designer who approves the model’s output isn’t deciding, they’re validating. The one who accepts a code change proposal isn’t writing, they’re curating. We’re becoming, slowly but not that slowly, ex-designers and ex-programmers. People who choose from a menu built by someone else. And choosing from someone else’s menu isn’t agency: it’s the most comfortable form of obedience.
The metacognitive laziness the studies measured and the hyperreality Baudrillard described are the same thing seen from two sides. On one side, the student turns in a text they don’t understand. On the other, the designer signs off on a result they didn’t decide on. In both cases the product is flawless and the process was left empty.
New modes of interaction, and taming the horse
As a technology, LLMs widen the range of ways to build and design. The problem isn’t the tool: it’s the default mode of using it, one that delivers a result and absorbs the process. The minimum bar needs to be different. There’s knowledge when there’s understanding of the tool and its uses, not simply when there’s a delivered output. A tool you don’t understand isn’t a tool: it’s a tacitly directed oracle. And you don’t learn from an oracle: you believe it and obey it, or you abandon it.
This is already happening with hardware. The cybersurgeons have been showing this for a while, with the idea of “low tech, high life.” The operation has two parts. The first is low tech: discarding what’s useless, weighing the real cost of each technology before adopting it, preferring the simple option when complexity adds nothing. The second is permacomputing: building and repairing what does work, extending the life of hardware, using only what’s necessary, treating computing the way permaculture treats the land. It’s not nostalgia or Luddism: it’s strategy, an attempt to decide, as much as possible (and as desirable).
Not long ago, a new way of thinking about building software with Language Models started circulating: “harness engineering.” For reasons of language and literary style, I’ll call it the Bridle. If the cybersurgeon decides what to repair and what to discard in hardware, the Bridle controls what’s allowed in and what gets tamed out of the Language Model’s ramblings—a runaway horse that easily loses its footing. Everything that isn’t the model itself is part of those limits, that control: the prompts, the tools it’s given, the memory that persists, the reasoning loops, what slice of the world enters as context and what’s kept out. All of this helps whoever is writing software structure how they use a Language Model so as not to lose the human role, or the learning, along the way. This is where human decision-making lives. The model, at least given the opaque tendencies of the most popular commercial models, remains a black box to the user, but you decide what to do once it opens. Sovereignty doesn’t mean not using the model: it means being the architect of the apparatus instead of a mere consumer.
And this isn’t just my intuition. There’s a place where it can actually be measured. SWE-bench is a benchmark that evaluates how well a system solves real software problems pulled from open repositories. When people started seriously examining what moved the needle, an uncomfortable fact emerged for anyone hoping everything gets solved by the next model: much of the improvement didn’t come from the model, it came from the scaffolding around the model. From the Bridle. Changing the structure that organized the code performed as well as, or better than, changing the model. It’s worth treating this cautiously and checking it properly before turning it into dogma, but the direction is fairly clear: how you structure the use matters as much as the raw power of what you’re using. Agency doesn’t live in the horse. It lives in the Bridle, and in its rider.
Your own tools and structures
I think it’s important to keep insisting that one of the best things we can do today is look for alternatives to how we build our tools, or, at the very least, be conscious of what and how we’re using them.
A while back I started proposing a mini-framework, four steps for creating prototypes that can iterate their way into full solutions. In short, it involves taking (1) a Context, in which certain (2) People take part, laying out a (3) Hypothesis for solving the problem, and choosing a set of (4) Expected Results to test it against. Put those four pieces together, and what you get is a way of building iterations of solution hypotheses. This can be applied, to a greater or lesser degree, to almost any situation. It’s obviously not the most exhaustive or definitive method—in fact it’s quite abstract and minimalist—but it gives you a better footing from which to define what you’re doing, and to make decisions.
I’m not proposing that we stop using Language Models. I’m far from a Luddite. I’m proposing something more uncomfortable: treating every use as a decision, not a reflex.
It doesn’t matter all that much whether the framework is this one or another one you use instead. What matters is that one exists: that between you and the Language Model there’s a structure you put there on purpose, rather than the default mode it ships with. That structure is, quite simply, the place where you become the one deciding again.
The window
I’m not proposing that we stop using Language Models. I’m far from a Luddite. I’m proposing something more uncomfortable: treating every use as a decision, not a reflex. Asking yourself, before delegating, whether you’ve ever done that by hand, whether you’ll be able to recognize when the result is yours and when it’s the average dressed up as an idea of your own. Without that memory of the craft, you can’t validate anything—let alone create it.
Ruocco closed his foundational piece with Hakim Bey’s Temporary Autonomous Zones: pockets of freedom outside the system’s gaze. I like to borrow that image and twist it a bit. The Temporary Autonomous Zones we have left are barely geographic anymore. They’re cognitive. They’re the few minutes when you think without an assistant, sketch without autocomplete, write without a model. Ever briefer. Ever more valuable. That window, for as long as we can manage to keep it open, is all the sovereignty there is.