Back to BERT in 2026: ModernGENA as a Strong, Efficient Baseline for DNA Foundation Models (opens in new tab)

Recent advances in DNA language models have mainly come from building larger and more complex architectures, making it harder to understand the effect of changes to standard components such as the transformer layers widely used in NLP. In this work, we study whether and how a modernized BERT-style backbone (ModernBERT) can be adapted to genomic sequence modeling to improve computational efficiency, training stability, and downstream performance. Under controlled experimental settings, we benc...