Inside Llama 3.1 405B MLPerf Training on Azure: System-Level Insights at 8K+ GPU Scale (opens in new tab)

By Shantanu Patankar, and Azin Heidarshenas As part of Azure's MLPerf Training v6.0 submission, we scaled Llama 3.1 405B (NVFP4) pretraining to 8,192 GB200 GPUs across 128 racks on Azure's Fairwater infrastructure, converging in 7.07 minutes (MLPerf Training Blog). Llama 3.1, with 405 billion active parameters, has the highest active parameter count among all models in the MLPerf suite. Getting a model of this scale to train efficiently requires a cascade of tightly coupled choices around par...