Abstract:We formalize a structural property of the causal (autoregressive) language modeling (CLM) objective: reversal invariance. Formally, the next-token prediction loss assigns identical likelihood to a corpus and its reversal, implying that standard CLM pretraining is direction-blind. This symmetry explains why models trained on reversed text can achieve comparable performance to those trained on forward text, despite the inherently time-asymmetric nature of human language and reasoning. We argue that this invariance represents a limitation of current pretraining objectives rather than a benign artifact. If natural language encodes directional dependencies - phonological…
Abstract:We formalize a structural property of the causal (autoregressive) language modeling (CLM) objective: reversal invariance. Formally, the next-token prediction loss assigns identical likelihood to a corpus and its reversal, implying that standard CLM pretraining is direction-blind. This symmetry explains why models trained on reversed text can achieve comparable performance to those trained on forward text, despite the inherently time-asymmetric nature of human language and reasoning. We argue that this invariance represents a limitation of current pretraining objectives rather than a benign artifact. If natural language encodes directional dependencies - phonological, morphological, or causal - a symmetric objective may fail to capture them. We therefore propose viewing pretraining through the lens of temporal asymmetry, motivating future work on loss functions and architectures that explicitly model the arrow of language while retaining standard language modeling capacity.
| Comments: | 7 pages, theoretical note |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2511.00341 [cs.CL] |
| (or arXiv:2511.00341v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2511.00341 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Mihir Sahasrabudhe [view email] [v1] Sat, 1 Nov 2025 00:51:46 UTC (16 KB)