395 words, 2 min read
⚠️ This post links to an external website. ⚠️
Back in 2022, my team and I were working on a service which was printing a fairly sizeable amount of logs from a distributed cluster of 20+ hosts: about 2-3 million log messages per hour in total. We were using Graylog, and querying those logs for an hour was taking no more than 1-3 seconds, so it was pretty quick.
Infra people hated Graylog though, since it required some annoying maintenance from them, and so at some point the decision was made to switch to Splunk instead. And when Splunk was finally rolled out, I had to find out that it was incredibly, ridiculously slow. Honestly, looking at it, I don’t quite understand how they are even selling it. If you’ve used Splunk, you might know that it has two modes…
395 words, 2 min read
⚠️ This post links to an external website. ⚠️
Back in 2022, my team and I were working on a service which was printing a fairly sizeable amount of logs from a distributed cluster of 20+ hosts: about 2-3 million log messages per hour in total. We were using Graylog, and querying those logs for an hour was taking no more than 1-3 seconds, so it was pretty quick.
Infra people hated Graylog though, since it required some annoying maintenance from them, and so at some point the decision was made to switch to Splunk instead. And when Splunk was finally rolled out, I had to find out that it was incredibly, ridiculously slow. Honestly, looking at it, I don’t quite understand how they are even selling it. If you’ve used Splunk, you might know that it has two modes: “Smart” and “Fast”. In “Smart” mode, the same query for an hour of logs was taking a few minutes. And in so called “Fast” mode, it was taking 30-60s (and that “Fast” mode has some other limitations which makes it a lot less useful). It might have been a misconfiguration of some sort (I’m not an infra guy so I don’t know), but no one knew how or wanted to fix it, and so it was clear that once Graylog is finally shut down, we’ll lose our ability to query logs quickly, and it was a massive bummer for us.
And I thought that it’s just ridiculous. 2-3M log messages doesn’t sound like such a big amount of logs, and it seemed like some old-school shell hacks on plain log files, without having any centralized logging server, should be about as fast as Graylog was (or at least, MUCH faster than Splunk), and it should be enough for most of our needs. Let me mention here that we weren’t using any containerization: the hosts were actual AWS instances running Ubuntu, and our backend was running there directly as systemd services, naturally printing logs to /var/log/syslog, so these plain log files were readily available to us.
And so that’s how the project started: I couldn’t stop thinking of it, so I took a week off, and went on a personal hackathon to implement a proof-of-concept log fetcher and viewer with a simple terminal UI, which is ssh-ing directly to the hosts, and analyzing plain log files using bash + tail + head + awk hacks.
If this post was enjoyable or useful for you, please share it! If you have comments, questions, or feedback, you can email my personal email. To get new posts, subscribe use the RSS feed.