As a best-selling author, I invite you to explore my books on Amazon. Don’t forget to follow me on Medium and show your support. Thank you! Your support means the world!
Let’s talk about building a system that can handle a constant flow of data. Imagine you’re trying to count cars on a busy highway in real-time. The cars keep coming, non-stop. You need to count them, maybe group them by color every minute, and do it all without your system falling over when traffic suddenly doubles. That’s the world of stream processing. In this article, I’ll show you how to build the core of such a system using Go. We’ll focus on three critical ideas: managing flow so we don’t get overwhelmed (backpressure), grouping data into time buckets (windowing), and doing it all reliably.
Go is a fantastic language for this. Its straightforward concurrency model with goroutines and channels feels like it was designed for data pipelines. You can think of a channel as a conveyor belt. One goroutine puts items on the belt, and another takes them off for processing. Our job is to build a network of these conveyor belts that can speed up, slow down, and organize items without breaking.
Let’s start with the very foundation: moving data from a source to a destination. A basic pipeline might look like this.
package main
import (
"fmt"
"time"
)
func main() {
// A source channel producing data
source := make(chan int)
// A sink channel receiving processed data
sink := make(chan int)
// Start a data producer
go func() {
for i := 1; i <= 5; i++ {
fmt.Printf("Producing: %d\n", i)
source <- i
time.Sleep(100 * time.Millisecond) // Simulate work
}
close(source)
}()
// Start a processing stage
go func() {
for value := range source {
result := value * 2 // Simple processing
sink <- result
}
close(sink)
}()
// Consume the final results
for result := range sink {
fmt.Printf("Result: %d\n", result)
}
}
This is a simple chain: produce, transform, consume. But what happens if the consumer is slower than the producer? The channel will fill up. In Go, a channel with a buffer can hold a certain number of items. Once it’s full, the producer will be blocked until space frees up. This is a primitive form of backpressure—the slow consumer indirectly slows down the fast producer. For a robust system, we need to manage this explicitly and gracefully.
A real-world source, like a sensor or a message queue, often produces data continuously. We need a way to represent this and control its speed.
type DataSource struct {
dataChan chan interface{}
rate int // desired messages per second
running bool
mu sync.RWMutex // protects the 'running' state
}