Local Gradient Accumulation Speeds Training 1.7 (opens in new tab)

PACI removes the bubbles that cripple asynchronous pipeline parallelism and shaves as much as 1.69× off time‑to‑accuracy compared with the fastest synchronous flush baseline. The paper demonstrates this gain on GPT‑2 Medium pre‑training while preserving the same peak memory usage. By locally accumulating gradients, PACI limits how far a micro‑batch can drift from the current weight version, so the pipeline stays fully busy without any global synchronization. Before PACI, the dominant strategy...