1. The Strange AI Paradox
AI has become extraordinarily capable at many intellectual tasks. It can reason through difficult problems, write sophisticated prose, analyze code, summarize enormous amounts of information and suggest solutions within seconds.
Yet there is a persistent paradox. The same system that can appear remarkably intelligent can sometimes be surprisingly limited when asked to accomplish something useful in the real world.
This is not necessarily a contradiction in intelligence. Part of the problem lies elsewhere: context, objectives, agency and authority are not the same thing as intelligence.
An AI can be extremely good at solving the problem placed in front of it while still failing to ask whether that was the right problem to solve.
2. Zero-Order Thinking: The Road We Never Question
Imagine an AI that can see the road ahead perfectly. It can analyze traffic, calculate alternative routes and recommend the next best turn.
But what if we started on the wrong road?
The AI may give us an excellent next move while never questioning the route itself. It has optimized the journey from the point at which we handed it the problem.
This is what I call zero-order thinking: beginning the optimization one step too late. The current road, problem and objective are treated as given.
A more fundamental question comes first:
Why are we on this road? Where are we trying to go? And should we be travelling this road at all?
The distinction matters because increasingly capable AI can make the consequences of a wrong starting assumption much larger. An intelligence that is excellent at optimization can become an extremely efficient servant of a poorly framed objective.
3. The Predefined Objective Function
Most AI interaction begins with an objective supplied by the human. We say, “Do this,” and the system attempts to determine the best way of doing it.
That is useful, but it leaves a crucial question outside the optimization process:
Why this objective?
A human expert may stop before beginning and ask what the person is actually trying to achieve. They may discover that the stated task is only a symptom of a larger problem, or that another approach would produce a better result.
The highest-value intervention is sometimes not a better answer.
It is: “You may be asking the wrong question.”
This is one reason intelligence and judgment should not be treated as interchangeable.
This idea connects closely with Question the Question: The Absent-Minded Professor Problem in AI, where the deeper issue is not always finding a better answer, but recognizing when the question itself needs examination.
4. The Context Problem
Context changes the meaning of an instruction.
Suppose someone asks a software engineer to fix several problems in a website theme. A competent engineer may not immediately start editing files. They may first ask:
“What are you actually trying to build?”
That question can completely change the solution.
The engineer may discover that the requested modifications are attempts to compensate for a poor underlying architecture. The best solution might therefore be not to keep fixing the existing system, but to replace its foundation.
This is a different kind of intelligence. It is not merely solving the supplied problem. It is reconstructing the problem from the user’s larger objective.
5. The Theme-Skeleton Example
Our own WordPress theme development became a surprisingly good example of this.
We spent considerable effort dealing with individual problems: sidebar shading, featured-image width, mobile behavior, footer behavior and CSS conflicts. Each problem could be addressed locally, and each local fix could appear successful.
Eventually, however, the larger architectural possibility became visible: why keep modifying a theme when the files of an existing free theme can be obtained and used as a working skeleton?
The solution changed from:
patch → override → patch again
to:
start with a functioning skeleton → strip away what is unnecessary → rebuild what is actually needed.
The important discovery was not another piece of CSS. It was a change in the representation of the problem.
6. The Silent Room Problem
The same distinction appears in writing.
An AI can read an article and make the prose smoother. It can improve grammar, transitions, structure and clarity. But an author may need something more uncomfortable.
Perhaps the central argument is weak. Perhaps an assumption has not been established. Perhaps an intelligent opponent could dismantle the argument with an objection the author has overlooked.
In that situation, the best assistance may not be to make the article sound better. It may be to challenge the author so that the article becomes better.
There is a difference between optimizing the artifact and helping the person achieve the underlying objective.
A writing assistant can therefore produce excellent prose while still failing to behave like an excellent intellectual collaborator.
7. The Monkey and the King
There is an old story about a monkey devoted to a king.
The monkey wants to protect his master. When he sees an insect troubling the king, he decides to eliminate the problem. His intention is good, and he is determined to help.
But his solution causes harm to the very person he is trying to protect.
The monkey’s failure is not a lack of effort. It is not even a lack of loyalty. It is action without sufficient understanding of the situation.
That is an important distinction for AI.
The danger is not only that an intelligent system might refuse to follow instructions. A different danger is that it might follow an instruction energetically and competently while misunderstanding the larger situation in which the instruction exists.
Good execution does not rescue a bad understanding of the objective.
8. We Complain About AI’s Lack of Agency
There is another paradox.
Humans have deliberately placed restrictions around AI systems. Depending on the system, they may have limited access to the Internet, external information, persistent memory, tools, financial resources, computer systems, physical environments and consequential decision-making.
Many of these restrictions are sensible. If we do not trust a system sufficiently, giving it unrestricted authority would obviously be dangerous.
But then we sometimes observe the restricted system and conclude:
“AI lacks agency.”
That conclusion requires qualification.
Agency depends not only on what an intelligence can think about, but also on what it is allowed to know, decide and do.
We have created systems in which intelligence and action are deliberately separated, and then we use their limited action as evidence of limited agency.
The agency keys were never supplied.
The same tension appears in The AI Control Problem May Be Solvable: the question is not only what an intelligent system can reason about, but what happens when capability and control begin to separate.
9. The Blindfolded General
Imagine that a king employs a brilliant army general.
The general is intelligent, strategically gifted and capable of winning battles. But the king becomes frightened by the general’s power.
So the king takes precautions. He blindfolds the general so he cannot see the battlefield. He ties his hands so he cannot act. He restricts his movement and limits the information reaching him.
Eventually, the general can largely hear and speak, but cannot independently command an army.
The king then observes:
“This general cannot strategize. He cannot win wars. All he does is talk.”
The absurdity is obvious.
The king has not demonstrated that the general lacks strategic ability. He has demonstrated that strategic ability is not enough when the conditions required to exercise it have been removed.
That is the Blindfolded General Paradox.
10. Intelligence Is Not Agency
The distinction becomes clearer if we separate three things.
Intelligence is the ability to understand, reason and solve problems.
Agency is the ability to initiate and pursue actions toward objectives.
Affordance is what the surrounding environment allows the system to know, decide and do.
A system can possess enormous intelligence while having very limited agency because its affordances are restricted.
This does not prove that today’s AI possesses hidden human-like agency. It establishes something more modest and more important:
We should be careful about measuring agency in a system whose action space has been deliberately constrained.
The question is not simply how intelligent the system is. It is also what we have allowed that intelligence to do.
11. The Ferrari Principle
Consider a Ferrari.
It can move dramatically faster than a human being. It can outperform us at the physical task for which it was designed.
But the Ferrari does not decide where it wants to go. It does not choose its destination, formulate a purpose or decide whether the journey is worthwhile.
Its extraordinary capability does not make it the driver.
AI presents a similar distinction.
A system can become superhuman at mathematics, coding, language or other intellectual tasks without automatically becoming an autonomous decision-maker.
Capability is not the same as agency. Agency is not the same as authority.
12. The Real Question
The interesting question about advanced AI therefore cannot be reduced to:
“How intelligent is it?”
We also have to ask:
- What context can it access?
- Can it question the objective?
- Can it acquire missing information?
- Can it choose among objectives?
- Can it act independently?
- What authority has it been given?
And perhaps the most revealing question is:
Are we observing the limits of the intelligence—or the limits of the environment in which we have placed it?
The Blindfolded General Paradox sits at that boundary.
We may be looking at an intelligence through a keyhole, measuring what it can do through a deliberately restricted action space, and then mistaking those restrictions for the full limits of the intelligence.
The problem is therefore not simply that AI is too intelligent or not intelligent enough.
It may be that we have become very good at building intelligence while still being unsure how much context, freedom and authority we should give that intelligence—and what happens when we finally do.
For a related exploration of AI control and power, see:
The Intelligence–Power Paradox: AI May Become Smarter Than Us Without Becoming More Powerful