Press enter or click to view image in full size

Photo by Steve Johnson on Unsplash

5 min read1 day ago

–

In a previous article, I compared the tech industry’s AI enthusiasm to a “cargo cult”. Organizations are rapidly launching AI initiatives and “transformation programs”, imitating the form of intelligence while often misunderstanding the statistical and associative nature of large language models. The belief that fluency implies understanding echoes the ritual of constructing runways in the jungle and waiting for airplanes that never arrive. [1]

But what if we’ve been looking at the metaphor backwards? What if the real cargo cultist isn’t the venture capitalist or the enterprise strategist, but the Large Language Model itself?

An LLM is an exceptionally skilled imitator. It generates text that resembles intelligence, creativity, or empathy – not by understanding concepts – but by modeling statistical relationships between tokens in its training data. It performs the form of language without engaging in meaning. [4][5] Its fluency can be striking, even emotionally compelling, yet it lacks grounding in perception, intention, or causal structure. Without access to real-world meaning, scaling alone does not guarantee understanding. The model performs its linguistic ritual perfectly, but the ritual does not produce cargo. [6][7]

The Perfect Ritual: How LLMs Master Form

The magic behind an LLM’s fluency is the Transformer architecture, which uses self-attention. In simple terms, as the model processes a token, it weighs the relevance of other tokens to build contextualized representations. [2][3] For instance, in “The animal didn’t cross the street because it was too tired,” self-attention helps relate “it” to “animal.” This is a brilliant feat of engineering, but it is still a high-dimensional statistical method. The model does not know what an animal is or what being tired feels like; it has learned statistical regularities from vast corpora. [4][5] It performs the ritual of language, not the act of meaning.

This is the core of the “stochastic parrots” critique: LLMs can stitch together well-formed linguistic sequences without reference to meaning. [5] They can assemble a convincing bamboo airplane from every blueprint they have ever seen, but there is no engine inside.

The Empty Cockpit: Three Signs the Planes Aren’t Landing

Hallucinations Are a Feature, Not a Bug

LLMs sometimes produce confident but false statements. Treating these “hallucinations” purely as bugs misses the point: as next-token predictors, models are optimized for plausible continuation, not truth-verification. When information is missing, they continue the ritual of answering. Grounding via retrieval or tools can reduce errors but cannot eliminate them so long as generation is driven by statistical prediction alone. [7]

Stuck on the First Rung of the Causal Ladder

Judea Pearl’s Ladder of Causation distinguishes association (seeing), intervention (doing), and counterfactuals (imagining). [8] Vanilla LLMs operate primarily at the associational level. Tool-augmented systems can simulate aspects of higher-rung reasoning (e.g., with planning modules or programmatic tools), but the language model proper remains fundamentally associative. [8][9]

“Emergence” Is Often a Measurement Mirage

Apparent “emergent abilities” have frequently turned out to be **evaluation artifacts: **Discontinuities induced by metrics or threshold effects rather than abrupt leaps in understanding. When measurement is improved, many jumps smooth out. This doesn’t make all non-linearities imaginary; it reframes them as scaling dynamics rather than sudden cognition. [6]

Why We Believe in the Ritual

LLM outputs exploit human tendencies to anthropomorphize. The **ELIZA effect, **first observed in the 1960s, showed how people attribute understanding to systems that mirror conversational form. [10][12] If something communicates like a human, we tend to treat it as if it were human.

Two cognitive shortcuts amplify this:

• Automation bias: we over-trust outputs from automated systems, especially when they are fluent and consistent. [13]
• Framing effects: confident, polished language sounds credible and can sway judgments even when facts are weak. [11]

An LLM can deliver a fabrication and a fact in the same authoritative tone; our cognition is predisposed to accept the ritual as reality. [11][12][13]

Living with the Perfect Imitator

Recognizing LLMs as “cargo cultists” is not a dismissal. It’s a call for what Richard Feynman termed utter honesty: use models for what they are, **masters of linguistic form, **and be explicit about what they are not. [1][4][5] They are not nascent minds or general reasoning engines; they are sophisticated pattern-matchers whose strengths shine in brainstorming, drafting, rewriting, summarization, and routine code scaffolding. Treat outputs as strong suggestions, not authoritative answers; apply grounding, verification, and human oversight where accuracy matters. [4][5][7][9]

The bamboo planes may never fly on their own. But if we stop waiting for magical cargo and instead treat LLMs as impressive, useful artifacts, we can build real value on the ground with clear eyes about their limits and strengths. [1]

I hope you found what you were looking for. Leave a clap if you liked it and don’t hesitate to share comments or corrections!

References

[1] Feynman, R. (1974). Cargo Cult Science. Caltech Commencement Address. [2] Vaswani, A., Shazeer, N., Parmar, N., et al. (2017). Attention Is All You Need. NeurIPS. [3] Alammar, J. (2018). The Illustrated Transformer. https://jalammar.github.io/illustrated-transformer/ [4] Bender, E., & Koller, A. (2020). Climbing Towards NLU: On Meaning, Form, and Understanding in LMs. ACL. [5] Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021). On the Dangers of Stochastic Parrots. FAccT. [6] Schaeffer, R., Miranda, B., & Koyejo, S. (2023). Are Emergent Abilities of Large Language Models a Mirage? arXiv:2304.15004. [7] Xu, Z., et al. (2024). Hallucination is Inevitable: An Innate Limitation of Large Language Models. arXiv:2401.11817. [8] Pearl, J., & Mackenzie, D. (2018). The Book of Why: The New Science of Cause and Effect. Basic Books. [9] Kıcıman, E., Zhang, M., Shah, S., et al. (2023). Causal Reasoning and Large Language Models: A Survey. arXiv:2305.07177. [10] Weizenbaum, J. (1966). ELIZA — A Computer Program for the Study of Natural Language Communication between Man and Machine. Communications of the ACM, 9(1), 36–45. [11] Tversky, A., & Kahneman, D. (1981). The Framing of Decisions and the Psychology of Choice. Science, 211(4481), 453–458. [12] Nielsen Norman Group. (2023). The ELIZA Effect and Anthropomorphism in AI. [13] Skitka, L. J., Mosier, K. L., & Burdick, M. (1999). Does Automation Bias Decision-Making? Human Factors, 41(4), 703–718.