Structure tokens sharpen the feature vocabulary of protein language models (opens in new tab)

Protein language models predict structure and function from amino acid sequences, but the internal computations that produce these predictions remain opaque. We applied sparse autoencoders to ESM-2 (650M parameters, sequence-only) and ESM-3 (1.4B parameters, multimodal) and found that 78% of learned features converge between the two architectures (permutation null: 14.2%, p < 0.001). These convergent features account for nearly all functional knowledge encoded by the models (functional site A...