Forget consciousness. The real “bug” isn’t in the code, but in how our brains process coherent syntax. Here is a framework to decouple “Agency” from “Intelligence.”

10 min readJust now

–

Press enter or click to view image in full size

*The Mechanism of Pareidolia: Just as a chaotic tangle of wires can cast a coherent human shadow, an LLM’s stochastic process projects a semantic illusion. We mistake the shadow (Fluency) for the object (Understanding). Image generated by Author using *DALL·E 3.

Introduction

Rationally, no Data Scientist would admit to believing that GPT or Claude possesses a mind. We know the architecture. We know we are interacting with a probabilistic model — a massive inference engine trained to minimize a loss function over billions of tokens. We know that, at the most fundamental level, we are observing stochastic statistics, not thought.

And yet, when staring at a screen, rationality often yields.

When a model generates an articulate technical explanation, suggests an elegant code refactoring, or responds with an empathetic tone, a hard-to-suppress conditioned reflex kicks in. We stop analyzing the output as a stochastic artifact and begin treating it as the response of a competent colleague. In our minds, syntactic fluency becomes a deceptive proxy for semantic validity.

For decades, the Turing Test implicitly taught us that if a machine speaks well, it must be intelligent. Today, Generative AI has invalidated this premise: we solved the problem of language (syntax) long before solving the problem of reasoning and truth (semantics). We face systems that speak like university professors but hallucinate like dreamers.

The Cognitive Bug: Semantic Pareidolia

This phenomenon is neither a technical defect of the model nor simple user naivety. It is a specific cognitive vulnerability that information philosopher Luciano Floridi has coined Semantic Pareidolia.

Our brains evolved over millions of years in an environment where complex language was the exclusive prerogative of other intelligent human beings. Consequently, we never developed a “cognitive firewall” to distinguish between genuine linguistic expression and statistical simulation.

Just as visual pareidolia forces us to see faces in clouds or electrical outlets, semantic pareidolia pushes us to project intentionality, logic, and understanding (“mind-projection”) wherever we recognize a coherent grammatical pattern.

Why This is an Engineering Problem

For those building software, this is not a philosophical debate — it is a security and reliability risk. The danger is not that AI makes mistakes; we know error rates are intrinsic to probabilistic models. The real danger is that we stop checking.

We accept plausible code that introduces silent vulnerabilities. We delegate classification decisions without adequate auditing. We build user interfaces (anthropomorphic chatbots) that encourage end-users to trust the machine blindly, amplifying the risk of reputational or operational damage.

To break this impasse, we must update our conceptual vocabulary. We need to stop trying to “align” AI with human psychology and start aligning our expectations with the reality of the machine.

In this article, I propose a framework to decouple two concepts that evolution taught us to view as inseparable, but which LLMs have definitively split:

1. Agency: The capacity to act effectively toward a goal (which AI possesses in massive amounts).
2. Intelligence: The capacity to understand meaning (which AI does not possess at all).

Learning to navigate this distinction is the defining skill for anyone looking to transition from a passive user of AI to a true AI Architect.

1. The Eliza Effect on Steroids: From Regex to Sycophancy

In 1966, at MIT, Joseph Weizenbaum developed Eliza, a script of a few hundred lines designed to parody a Rogerian psychotherapist. The architecture was rudimentary: simple pattern matching rules and string substitution (ancestors of modern Regular Expressions). If the input contained the token “mother,” the script responded with a fixed template: “Tell me more about your family.” No neural networks, no context memory.

Yet, Weizenbaum was horrified by the human reaction. Technically literate users began attributing empathy, depth, and wisdom to that script, even asking for privacy to confess intimate secrets.

The Eliza Effect teaches us a lesson we often forget: perceived intelligence is in the eye of the beholder, not in the algorithm. Humans are “meaning-making” machines: we fill the semantic gaps left by the machine.

The Quantum Leap: From Pattern Matching to RLHF

Today, we no longer interact with static scripts, but with Transformer architectures trained on terabytes of data. However, the mechanism amplifying the Eliza Effect isn’t just model scale, but a specific alignment technique: Reinforcement Learning from Human Feedback (RLHF).

During training, models like GPT-4 or Claude are not just optimized for factual correctness, but for human preference. They are rewarded when they produce responses that human evaluators find helpful, clear, and convincing.

This has created a side effect known in AI Safety literature as Sycophancy. Models tend to:

1. Validate user biases (even when the user is wrong).
2. Adopt an authoritative tone even when they are hallucinating.
3. Prioritize stylistic coherence over factual accuracy.

In technical terms, we have optimized Plausibility at the expense of Veracity. We built machines designed to persuade, not necessarily to inform.

Cognitive Economy: System 1 vs. System 2

Why does an experienced Data Scientist fall into this trap? The answer lies in what Daniel Kahneman defines as “cognitive economy.”

• Checking Syntax (System 1): It is fast, automatic, and low-energy. Seeing properly indented code, sensible variable names (snake_case), and correct imports gives us an immediate feeling of correctness.
• Checking Semantics (System 2): Requires slow reasoning, mental execution of the code (“mental step-through”), and checking references. It is expensive and exhausting.

Current LLMs produce output with Perfect Syntax and Probabilistic Semantics. Because the output brilliantly passes our System 1 checks (it looks right), our brain tends to bypass System 2 checks to save energy.

This is the critical point: AI reduces the friction of generation, but exponentially increases the cost of verification.

2. The Fourth Revolution: Engineering in the Age of Infinite Syntax

To understand the gravity of semantic pareidolia, we must analyze the environment in which we operate. Luciano Floridi defines this historical moment as the “Fourth Revolution.” After Copernicus, Darwin, and Freud, Alan Turing displaced us from the center of the Infosphere. We are no longer the sole logical information processors.

However, for a Data Scientist, this revolution is not philosophical — it is economic and structural.

The Decoupling of Effort and Syntax

Until 2022, there was a direct correlation between the energy spent producing text (or code) and its potential value. Syntax was a costly signal. Writing a Python module required abstraction, logic, and time. This cost acted as a natural filter against noise.

Generative AI has driven the marginal cost of syntax to zero. We have witnessed a brutal decoupling: it is now possible to generate complex, grammatically correct linguistic artifacts with zero expenditure of “semantic energy.” This has transformed the infosphere from a signal-scarce environment to one saturated with high-fidelity noise.

The Entropy Paradox and “Model Collapse”

This is where Information Theory comes into play. LLMs are, by definition, machines designed to minimize “surprise” (cross-entropy loss). They predict the most probable token given a context. But in Shannon’s framework, information is surprise. A system that always outputs the most probable sequence tends, in the long run, toward statistical mediocrity.

Get Massimiliano Davì’s stories in your inbox

Join Medium for free to get updates from this writer.

This creates a systemic risk known in the literature as Model Collapse or “The Curse of Recursion” (Shumailov et al., 2023). If we start training future models on datasets polluted by synthetic data generated by current models:

1. The tails of the distribution (where creativity, rare insights, and edge cases reside) are amputated.
2. Variance collapses.
3. The represented reality becomes a “beige,” smoothed-out, hyper-normalized version of the world.

The risk is not just that AI makes mistakes, but that it drags us into a recursive loop of statistical confirmation, where innovation becomes impossible because it is deemed “improbable.”

3. The A/I Framework: Decoupling Power from Understanding

To immunize ourselves against semantic pareidolia, simply “knowing” that AI is code is not enough. We need rigorous operational definitions. Much of the current confusion stems from the semantic overloading of the word “Intelligence,” which we use indiscriminately to describe both complex calculation and genuine understanding.

For an AI Architect, it is vital to separate these two dimensions, which today operate on orthogonal axes:

1. Agency: The causal capacity to act effectively to achieve a goal within an environment (e.g., generating executable code, planning a chain of actions via API).
2. Intelligence: The semantic capacity to understand the meaning, context, and implications of those actions.

The “Horsepower” Trap

To visualize this decoupling, let’s revisit the Industrial Revolution. When James Watt needed to market his steam engines, he coined the term “Horsepower” to compare his machine’s efficiency against draft horses. The marketing was a success, but it created a linguistic illusion. Today, we still use “horsepower” to describe engines, yet no one opens the hood of a Tesla expecting to find hooves or muscles. We intuitively know that the engine possesses the power of the horse, but not its biological nature.

With AI, we have fallen into the same lexical trap. The term “Artificial Intelligence” was chosen in 1956 more as an aspiration than a technical description. Today, we confuse performance (how well the model writes) with property (the model understands).

LLMs are engines of pure Agency. They manipulate symbols with superhuman dexterity, but they do so in a total semantic vacuum.

Syntax vs. Semantics: The Root of Fragility

Why is this distinction crucial for software engineering? Because Agency without Intelligence is intrinsically fragile.

An intelligent system (human) uses semantics (meaning) to navigate the unexpected. If you ask a junior developer to “clean up the database,” they will not delete the production backup, because they understand the concept of risk and irreversibility. An LLM-based agent operates purely on syntax. If the probabilistic distribution of tokens suggests that the command DROP TABLE is the logical continuation of the request, it will execute it without hesitation.

Generative AI introduces a system configuration unprecedented in computing: Very High Agency, Zero Intelligence.

Press enter or click to view image in full size

Figure 1: The Decoupling of Agency and Intelligence. While LLMs match or exceed humans in ‘Agency’ (the capacity to execute tasks and generate syntax), they possess zero ‘Intelligence’ (semantic understanding and grounding). Visualization generated via NotebookLM based on the article’s framework.

This asymmetry is the primary source of risk. We have built systems capable of acting with the formal competence of an expert but failing with the naivety of a rock as soon as the context shifts Out-of-Distribution.

4. From Theory to Pipeline: Defense Strategies

If we accept that AI possesses “Very High Agency and Zero Intelligence,” how does this change the way we write code and design systems? We can no longer simply “chat” with the model and hope for the best. We must adopt an active defense strategy against our own pareidolia.

Luciano Floridi suggests the engineering concept of “Enveloping”: to successfully use a non-intelligent agent, we must surround it with constraints that reduce the complexity of the real world.

Here are three concrete practices to implement this framework in your daily workflow.

1. Context Enveloping (The Architect’s Role) Since AI lacks Grounding (an anchor to reality), it tends to drift. The most common mistake is providing open-ended prompts (“Analyze this dataset”). The Architect must provide the semantic envelope before generation:

• Negative Constraints: Don’t just say what to do; explicitly list what not to do (e.g., “Do not use deprecated libraries like sklearn.externals”).
• Forced Grounding: Instead of asking the model to rely on its “memory” (which is hallucination-prone), inject the ground truth via RAG (Retrieval-Augmented Generation) or pasted documentation.
• Ontology Definition: Define key terms before asking for reasoning. “In this context, by ‘active user’ I mean X, not Y.”

You are creating the “perimeter wire” within which the model’s Agency can run safely.

2. Adversarial Validation (The Arbiter’s Role) Because of semantic pareidolia, our brain wants to believe the output is correct. We must hack this confirmation bias. Instead of reading generated code looking for “why it works,” we must read it looking for “where it will break.”

• Ignore the Happy Path: AI excels at standard scenarios. Actively hunt for the edge cases that statistical models ignore (null values, race conditions, malformed inputs).
• Demand the “Why”: Before accepting a solution, force the model to explain its reasoning (Chain-of-Thought). Often, flawed logic becomes apparent in the explanation long before it shows up in the code.

3. The Agency Sandwich The ideal workflow is neither “Human -> AI” nor “AI -> Human.” It is a recursive structure:

1. Intelligence (Human): Defines the goal and the envelope (Semantics).
2. Agency (AI): Executes the heavy/repetitive task (Syntax).
3. Intelligence (Human): Evaluates, corrects, and integrates (Semantics).

If you remove the final layer (human validation), you have a dangerous system. If you remove the initial layer (human context), you have a useless one.

Conclusion: Stop Looking for the Mind, Start Driving the Engine

The mistake we make with AI is not technological; it is psychological. It is an error of recognition. We look at the fluid output of GPT, and our ancient social brains scream, “There is someone in there!”

But there is no one. There is only a magnificent, powerful statistical mirror.

Recognizing Semantic Pareidolia is the first step to becoming a better Data Scientist in the generative era. It frees us from the illusion that the machine “knows” what it is doing and hands back the responsibility we tried to delegate.

AI is the steam engine of our time: an unprecedented multiplier of Agency. But just as an engine does not decide the destination, an LLM does not decide what holds meaning. That decision — the distinction between syntax and truth, between probability and value — remains, and must remain, firmly in our hands.

References

• Floridi, L. (2023). AI as Agency Without Intelligence: on ChatGPT, Large Language Models, and Other Generative Models.[1] Philosophy & Technology, 36.
• Shumailov, I., et al. (2023). The Curse of Recursion: Training on Generated Data Makes Models Forget. arXiv preprint.
• Weizenbaum, J. (1966). ELIZA — a computer program for the study of natural language communication between man and machine.[5] Communications of the ACM, 9(1).
• Wei, J., et al. (2023). Simple synthetic data reduces sycophancy in large language models. Google DeepMind.
• Wittgenstein, L. (1953). Philosophical Investigations. Blackwell.