At the beginning of 2024, I made what felt like a simple New Year’s resolution:

"This year, I will build my own language model from scratch."

Like many engineers, I had spent years working with language models, using APIs, training small versions, and reading research papers. But deep inside, I wanted to understand them at the most fundamental level.

Not just how to use them, but how to build one.

I had no idea what I was getting myself into.

Where to find the Book

• Gumroad: https://plakhera.gumroad.com/l/BuildingASmallLanguageModelfromScratch
• Leanpub: https://leanpub.com/buildingasmalllanguagemodelfromscratch

The Problem No One Talks About

When I started, I realized something almost immediately: learning to build an LLM is far harder than it appears online.

The resources fell into two extremes:

• Tutorials that were so basic they skipped the real challenges
• Research papers that assumed you already had a PhD in deep learning

Some assumed you knew neural networks.

• Others assumed you knew PyTorch.
• Most couldn’t be followed end-to-end without filling massive gaps yourself.

And almost none of them talked about the actual journey, the debugging, the failures, the dead ends, the experiments that didn’t work, the GPUs that crashed at 2 a.m.

Everyone showed the final model. No one showed the road that led to it.

Eighteen Months of Building, Breaking, and Starting Over My journey was not linear.

• I quit the project at least five times.
• I also restarted it five times.

But each failure taught me something new:

• that GPU memory is a world of its own
• that dataset quality matters more than anything
• that model architecture only "clicks" when you understand every line
• that deep learning frameworks hide complexity you must eventually uncover

Slowly, painfully, things started making sense. Not because I mastered theory, but because I lived through the mistakes. And that’s when the idea struck me: If beginners are struggling because no one explains things from scratch… why not write the book I wish someone had written for me?

A Book Built on Experience: Not Assumptions And that’s how Building Small Language Models from Scratch was born.

• It is not a research paper.
• It is not a high-level summary.
• It is not a collection of bullet points or diagrams.

It is the complete journey, written the way real engineers learn.

• I assume nothing, not even that you know PyTorch or how a neural network works.
• I explain every single line of code.
• I walk through the architecture step by step.

I dive into topics most books ignore, including:

• GPU fundamentals
• Data collection and cleaning
• KV Cache
• Multi-Query and Grouped Query Attention
• Quantization
• Mixture of Experts
• Rotary Position Embedding
• RMSNorm
• SwiGLU

And much more.

The final book… ended up being 854 pages. Not because I planned it that way, but because that’s how long the real story required.

Why I’m Sharing This Now

If you have ever wanted to build a language model… If you’ve tried learning from the internet but felt lost… If you’re someone who needs more than theory to stay motivated… This book is written for you. Because the truth is simple:

The hardest part about building LLMs is not the math or the code, it’s finding a guide that doesn’t leave you behind.

✅GitHub (examples and code): https://github.com/ideaweaver-ai/Building-Small-Language-Model-from-Scratch-A-Practical-Guide-Book

✅YouTube overview of the book: https://www.youtube.com/watch?v=5CbsN2dtQk0