Go & Backend November 16, 2025 8 min read

Production bug from lost orders. One-line fix with atomic operations. Complete guide to Go’s memory model without fluff.

Key Takeaways

• Happens-Before Relationships: Memory visibility requires proper synchronization through channels, mutexes, or atomic operations
• Race Conditions: Unsynchronized concurrent access is undefined behavior - always use go test -race
• Atomic Operations: Provide memory barriers and prevent CPU reordering issues
• Channel Semantics: Send happens-before receive, close happens-before receive from closed channel
• Lock Ordering: Always acquire locks in consistent order to prevent deadlocks

Table of Contents

1. Lost Orders: How It Started
2. Why Understanding the Memory Model Matters
3. A Small Real Example
4. Three Tools That Make Your Code Predictable
5. A Single Example That Explains the Model
6. A Minimal Lock-Free Example
7. A Practical Pattern You Need in Almost Any Service
8. A Tiny Test Every Project Should Have
9. The Main Takeaway
10. If You Want to Dig Deeper

Lost Orders: a short story of how it all started

In one of our services, we had a background process that collected results from workers and periodically flushed them. In tests — perfect. On staging — fine. In production — every few hours we discovered that some processed tasks simply disappeared.

No errors. No warnings. Just silence — and holes in the data.

The root cause turned out to be painfully simple: a regular counter was being updated concurrently without synchronization. Sometimes operations overtook each other, sometimes increments were lost. Under real load, this turned into financial discrepancies.

The fix was a single line (atomic.AddInt64), but the lesson was far more important: without a happens-before relationship between goroutines, nothing behaves the way you intuitively expect.

---

1. Why understanding the memory model matters

Almost every serious bug in concurrent code boils down to one question:

“I wrote the value. Why didn’t the other goroutine see it?”

The answer is almost always the same: the data was written, but not published.

Meaning:

• you changed something,
• but the compiler or CPU reordered operations,
• and the other goroutine saw stale data.

Go’s memory model is strict: without happens-before, visibility is not guaranteed.

This is invisible on small projects. Under load, it becomes a real problem.

---

2. A small real example

Multiple workers wrote results into a shared structure. Another goroutine periodically flushed those results. A counter tracked how many jobs had been processed.

type Results struct {
items []Item
count int64  // problem here
}

func (r *Results) Add(item Item) {
r.items = append(r.items, item)
r.count++  // read + write, not atomic
}

No synchronization.

On small loads — fine. Under production load — missing data.

Why? Because count++ isn’t atomic — it’s three operations:

1. Read current value
2. Add 1
3. Write back

Between those operations, another goroutine could sneak in and do the same. Both read the same value, both increment, both write back — one increment is lost.

The fix:

func (r *Results) Add(item Item) {
r.mu.Lock()
r.items = append(r.items, item)
r.count++
r.mu.Unlock()
}

Or if you only need the counter:

atomic.AddInt64(&r.count, 1)

Invisible in tests. Deadly under load.

---

3. Three tools that make your code predictable

Go gives a few simple primitives that guarantee ordering.

3.1 Channels

Send happens-before receive.

ch := make(chan int)
var data string

go func() {
data = "hello"
ch <- 1
}()

<-ch
fmt.Println(data) // always "hello"

Channels are one of the safest ways to publish data.

3.2 Mutexes

Unlock in one goroutine happens-before Lock in another.

Reliable, simple, rarely slow.

3.3 Atomics

Atomics define strict memory barriers: what comes before cannot be moved after, and vice versa.

They don’t “protect” data — but they guarantee visibility.

---

4. A single example that explains the model

The classic broken pattern:

type Message struct {
data  string
ready uint32
}

func (m *Message) unsafePublish(data string) {
m.data = data
m.ready = 1 // can be reordered
}

The CPU is allowed to reorder these operations. Why?

The processor sees: data and ready are different memory addresses. The operations are independent. For performance, it can reorder them.

Result: another goroutine reads ready == 1, but data is still stale.

Correct version:

func (m *Message) safePublish(data string) {
m.data = data
atomic.StoreUint32(&m.ready, 1)
}

func (m *Message) safeRead() string {
if atomic.LoadUint32(&m.ready) == 1 {
return m.data
}
return ""
}

atomic.StoreUint32 creates a memory barrier. Everything before it stays before it. The other goroutine sees consistent state.

One line makes the difference.

---

5. A minimal lock-free example

Lock-free structures look magical, but they only rely on the same memory model rules.

type node struct {
next unsafe.Pointer
val  int
}

type Stack struct {
head unsafe.Pointer
}

func (s *Stack) Push(v int) {
n := &node{val: v}
for {
head := atomic.LoadPointer(&s.head)
n.next = head
if atomic.CompareAndSwapPointer(&s.head, head, unsafe.Pointer(n)) {
return
}
}
}

func (s *Stack) Pop() (int, bool) {
for {
head := atomic.LoadPointer(&s.head)
if head == nil {
return 0, false
}
n := (*node)(head)
if atomic.CompareAndSwapPointer(&s.head, head, n.next) {
return n.val, true
}
}
}

This works because Go guarantees ordering around atomic operations. CAS creates a full memory barrier.

---

6. A practical pattern you need in almost any service

A worker pool with proper shutdown.

type Pool struct {
jobs chan int
quit chan struct{}
}

func (p *Pool) worker() {
for {
select {
case j := <-p.jobs:
_ = j
case <-p.quit:
return
}
}
}

Channels guarantee publication of values and clean shutdown semantics.

---

7. A tiny test every project should have

Race detector is your friend.

func TestRace(t *testing.T) {
var x int
done := make(chan bool)

go func() {
x++
done <- true
}()
go func() {
x++
done <- true
}()

<-done
<-done
}

Run:

go test -race

The race detector instruments your code and tracks all memory accesses. It will catch data races that only appear under specific timing conditions.

If you don’t write such tests — you will find the races in production.

---

8. The main takeaway

The Go memory model boils down to one simple rule:

If two goroutines access the same memory concurrently and there’s no synchronization, Go guarantees nothing.

This isn’t a compiler flaw. It’s how every fast system works.

Synchronization isn’t overhead. It’s control.

---

9. If you want to dig deeper

The official Go memory model documentation is here: https://go.dev/ref/mem

For practical examples, including lock-free structures, race tests, and advanced patterns, check out the Go source code itself — particularly sync/atomic and sync package tests.