https://www.together.ai/blog/decentralized-training-of-foundation-models-in-heterogeneous-environments (opens in new tab)

--- description: Training foundation models typically requires expensive dedicated clusters. This research explores leveraging decentralized, heterogeneous compute with slower interconnects. title: Decentralized training of foundation models in heterogeneous environments --- ⚡️ FlashAttention-4: up to 1.3× faster than cuDNN on NVIDIA Blackwell → Introducing Together AI's new look → 🔎 ATLAS: runtime-learning accelerators delivering up to 4x faster LLM inference → ⚡ Together GPU Clusters: s...