WeChat AI just dropped a paper called Continuous Autoregressive Language Models (CALM),it basically rethinks how LLMs generate text. Instead of predicting one token at a time from a discrete vocabulary (the slow, softmax-heavy way every GPT-style model works), CALM predicts continuous vectors that each represent multiple tokens.
These vectors are learned through a high-fidelity autoencoder that can compress, say, 4 tokens into one latent vector and reconstruct them with over 99.9% accuracy. So the model generates “semantic chunks” instead of words, cutting generation steps by 4× while keeping meaning intact.
Because the model operates in continuous space, there’s no softmax, no cross-entropy, and no perplexity.
Training uses an energy...
WeChat AI just dropped a paper called Continuous Autoregressive Language Models (CALM),it basically rethinks how LLMs generate text. Instead of predicting one token at a time from a discrete vocabulary (the slow, softmax-heavy way every GPT-style model works), CALM predicts continuous vectors that each represent multiple tokens.
These vectors are learned through a high-fidelity autoencoder that can compress, say, 4 tokens into one latent vector and reconstruct them with over 99.9% accuracy. So the model generates “semantic chunks” instead of words, cutting generation steps by 4× while keeping meaning intact.
Because the model operates in continuous space, there’s no softmax, no cross-entropy, and no perplexity.
Training uses an energy-based objective that compares predicted vs. real vectors, and evaluation uses a new metric called BrierLM, a likelihood-free stand-in for perplexity. In benchmarks on The Pile and WikiText-103, CALM matched or beat standard Transformers with ~40% less compute. It’s not just a speed trick, it’s a new scaling direction: instead of making models bigger, make each generative step carry more meaning.
Paper : https://arxiv.org/abs/2510.27688
Explanation : https://youtu.be/tLWBzya9dwA?si=k-9ozLk_PvU-V6au