From 24 Hours to 2 Hours: How We Fixed a Broken BI System With Apache Airflow (opens in new tab)

The System Was Broken, and Everyone Knew It Our dashboards refreshed overnight. That was the expectation. Then, one week, they started taking six hours. Then eight. On a bad day, the full 24 hours. Business users would come in on Monday morning and still see Friday's numbers. The data was wrong, too. Not wrong in an obvious way. Wrong in the quiet way where someone in finance notices a number looks off, checks it manually, finds a discrepancy, and then stops trusting the system. That is the w...