Beyond Stationarity: Rethinking Codebook Collapse in Vector Quantization (opens in new tab)

arXiv:2602.18896v1 Announce Type: new Abstract: Vector Quantization (VQ) underpins many modern generative frameworks such as VQ-VAE, VQ-GAN, and latent diffusion models. Yet, it suffers from the persistent problem of codebook collapse, where a large fraction of code vectors remains unused during training. This work provides a new theoretical explanation by identifying the nonstationary nature of encoder updates as the fundamental cause of this phenomenon. We show that as the encoder drifts, ...