Towards million-token context windows: a topology-preserving framework for adaptive transformer sparsification (opens in new tab)

Transformer self-attention and billion-node network analyses share a key limitation: all-to-all evaluation creates an $$O(N^2)$$ computational cost. Existing methods address this by either distributing the workload across hardware or substituting recurrent operators. This trades associative recall for efficiency. We present Reduced Interaction Sampling (RIS), a stochastic sparsification framework. RIS computes only a fraction of possible pairwise interactions. By leveraging topological redund...