Title:Mechanistic Interpretability of Antibody Language Models Using SAEs
Abstract:Sparse autoencoders (SAEs) are a mechanistic interpretability technique that have been used to provide insight into learned concepts within large protein language models. Here, we employ TopK and Ordered SAEs to investigate an autoregressive antibody language model, p-IgGen, and steer its generation. We show that TopK SAEs can reveal biologically meaningful latent features, but high feature concept correlation does not guarantee causal control over generation. In contrast, Ordered SAEs impose an hierarchical structure that reliably identifies steerable features, but at the expense of more com…
Title:Mechanistic Interpretability of Antibody Language Models Using SAEs
Abstract:Sparse autoencoders (SAEs) are a mechanistic interpretability technique that have been used to provide insight into learned concepts within large protein language models. Here, we employ TopK and Ordered SAEs to investigate an autoregressive antibody language model, p-IgGen, and steer its generation. We show that TopK SAEs can reveal biologically meaningful latent features, but high feature concept correlation does not guarantee causal control over generation. In contrast, Ordered SAEs impose an hierarchical structure that reliably identifies steerable features, but at the expense of more complex and less interpretable activation patterns. These findings advance the mechanistic interpretability of domain-specific protein language models and suggest that, while TopK SAEs are sufficient for mapping latent features to concepts, Ordered SAEs are preferable when precise generative steering is required.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Quantitative Methods (q-bio.QM) |
| Cite as: | arXiv:2512.05794 [cs.LG] |
| (or arXiv:2512.05794v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2512.05794 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Rebonto Haque [view email] [v1] Fri, 5 Dec 2025 15:18:50 UTC (5,593 KB)