The real AI risk isn't hallucinations. It's forgetting how to think.

A post from Ergosphere, written by a research educator, makes a distinction that is easy to agree with in principle and easy to ignore in practice. The argument is not that AI tools produce bad outputs. It is that two students can produce identical outputs via very different paths, and only one of those paths builds the capability to do the work independently next time. Alice struggled through the project, got confused, worked through the confusion, and now has something she cannot easily describe but genuinely possesses. Bob used an agent, produced the same paper in a fraction of the time, and has nothing except the output. Institutionally they look the same. Practically, they are not.

The framing that makes this more than a standard “learn the fundamentals” argument is the concept of the supervision paradox. When a physics instructor describes using AI to run experiments and getting plausible-looking but incorrect results, the natural response is “so supervise the model more carefully.” But the supervision itself is the learning. The process of checking whether the model’s answer is right, knowing what a plausible-but-wrong answer looks like versus a correct one, having the intuition to catch the subtle error in step three of a six-step derivation: all of that comes from having done the work manually first. A stronger model does not solve this. It makes the plausible-but-wrong outputs more convincing.

This has a specific texture in software. There is a clear difference between asking a model to generate boilerplate for a library you understand well, and asking a model to design a system you are trying to understand by watching what it generates. The first is leverage. The second is a particularly efficient way to stay confused. The danger Ergosphere is pointing at is not that you use AI for the first category. It is that the boundary between the two categories gets blurry when outputs are good enough that you stop noticing you have crossed it.

Academic and corporate incentive structures reward output, not understanding. When producing a technically correct deliverable using an AI agent is faster than producing the same deliverable by understanding the domain, rational actors will optimise for the faster path. This is not a character flaw. It is a response to how systems are structured. The consequences accumulate slowly: individual practitioners who can operate AI tools competently but cannot do independent analysis when the task falls outside the model’s training distribution.

The practical implication is not “use AI less.” It is “be deliberate about which parts of your work are load-bearing for your own development.” If you are trying to build capability in a domain, working through confusion without immediately delegating it to a model is probably necessary. If you are applying capability you already have, delegation is fine. The uncomfortable part is that this requires accurate self-assessment of where your actual knowledge boundaries are, which is exactly the kind of thing that comfortable drift erodes.