Title:Compute Trends Across Three Eras of Machine Learning
Abstract:Compute, data, and algorithmic advances are the three fundamental factors that guide the progress of modern Machine Learning (ML). In this paper we study trends in the most readily quantified factor - compute. We show that before 2010 training compute grew in line with Moore’s law, doubling roughly every 20 months. Since the advent of Deep Learning in the early 2010s, the scaling of training compute has accelerated, doubling approximately every 6 months. In late 2015, a new trend emerged as firms developed large-scale ML models with 10 to 100-fold larger requirements in training compute. Based on these observations we split the history of compute in ML into three e…
Title:Compute Trends Across Three Eras of Machine Learning
Abstract:Compute, data, and algorithmic advances are the three fundamental factors that guide the progress of modern Machine Learning (ML). In this paper we study trends in the most readily quantified factor - compute. We show that before 2010 training compute grew in line with Moore’s law, doubling roughly every 20 months. Since the advent of Deep Learning in the early 2010s, the scaling of training compute has accelerated, doubling approximately every 6 months. In late 2015, a new trend emerged as firms developed large-scale ML models with 10 to 100-fold larger requirements in training compute. Based on these observations we split the history of compute in ML into three eras: the Pre Deep Learning Era, the Deep Learning Era and the Large-Scale Era. Overall, our work highlights the fast-growing compute requirements for training advanced ML systems.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computers and Society (cs.CY) |
| Cite as: | arXiv:2202.05924 [cs.LG] |
| (or arXiv:2202.05924v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2202.05924 arXiv-issued DOI via DataCite | |
| Journal reference: | 2022 International Joint Conference on Neural Networks (IJCNN), Padua, Italy, 2022, pp. 1-8 |
| Related DOI: | https://doi.org/10.1109/IJCNN55064.2022.9891914 DOI(s) linking to related resources |
Submission history
From: Jaime Sevilla [view email] [v1] Fri, 11 Feb 2022 22:42:47 UTC (2,618 KB) [v2] Wed, 9 Mar 2022 19:30:21 UTC (1,197 KB)