One Slow DDP Rank Can Hold Back Your Whole PyTorch Job (opens in new tab)

A PyTorch DDP job can be slow without looking broken. No crash. No OOM. All GPUs are doing something. The training loop just takes longer…