Small is the new large

I’ve been talking to people about small language models (SLMs) for a little while now. They’ve told me they’ve got great results and they’re saving money compared to using LLMs; these are people running businesses so they know what they’re talking about. At an AI event, someone recommended I read the recent and short NVIDIA SLM paper, so I did. The paper was compelling; it gave the simple message that SLMs are useful now and you can save time and money if you use them instead of LLMs.

(If you want to use SLMs, you’ll be using Ollama and HuggingFace. They work together really well.)

As a result of what I’ve heard and read, I’ve looked into SLMs and I’m going to share with you what I’ve found. The bottom line is: they’re worth using, but with strong caveats.

What is a SLM?

The boundary between an SLM and an LLM is a bit blurry, but to put it simply, an SLM is any model small enough to run on a single computer (even a laptop). In reality, SLMs require quite a powerful machine (developer spec) as we’ll see, but nothing special, and certainly nothing beyond the budget of almost all businesses. Many (but not all) SLMs are open-source.

(If your laptop is "business spec", e.g., a MacBook Air, you probably don’t have enough computing power to test out SLMs.)

How to get started

To really dive into SLMs, you need to be able to use Python, but you can get started without coding. Let’s start with the non-coders path because this is the easiest way for everyone to get going.

The first port of call is visiting ollama.com and downloading their software for your machine. Install the software and run it. You should see a UI like this.

Out-of-the-box, Ollama doesn’t install any SLMs, so I’m going to show you how to install a model. From the drop down menu on the bottom right, select llama3.2. This will install the model on your machine which will take a minute or so. Remember, these models are resource hogs and using them will slow down your machine.

Once you’ve installed a model, ask it a question. For example, "Who is the Prime Minister of Canada?". The answer doesn’t really matter, this is just a simple proof that your installation was successful.

(By the way, the Ollama logo is very cute and they make great use of it. It shows you the power of good visual design.)

So many models!

The UI drop down list shows a number of models, but these are a fraction of what’s available. Go to this page to see a few more: https://ollama.com/library. This is a nice list, but you actually have access to thousands more. HuggingFace has a repository of models that follow the GGUF format, you can see the list here: https://huggingface.co/models?library=gguf.

Some models are newer than others and some are better than others at certain tasks. HuggingFace have a leaderboard that’s useful here: https://huggingface.co/spaces/ArtificialAnalysis/LLM-Performance-Leaderboard. It does say LLM, but it includes SLMs too and you can select just a SLM view of the models. There are also model cards you can explore that give you insight into the performance of each model for different types of tasks.

To select the right models for your project, you’ll need to define your problem and look for a model metric that most closely aligns with what you’re trying to do. That’s a lot of work, but to get started, you can install the popular models like mistral, llama3.2, and phi3 and get testing.

Who was the King of England in 1650?

You can’t just generically evaluate an SLM, you have to evaluate it for a the task you want to do. For example, if you want a chatbot to talk about the stock you have in your retail company, it’s no use testing the model on questions like "who was King of England in 1650?". It’s nice if the model knows Kings & Queens, but not really very useful to you. So your first task is defining your evaluation criteria.

(England didn’t have a King in 1650, it was a republic. Parliament had executed the previous King in 1649. This is an interesting piece of history, but why do you care if your SLM knows it?)

Text analysis: data breaches

For my evaluation, I chose a project analyzing press reports on data breaches. I selected nine questions I wanted answers to from a press report. Here are my questions:

• "Does the article discuss a data breach - answer only Yes or No"
• "Which entity was breached?"
• "How many records were breached?"
• "What date did the breach occur - answer using dd-MMM-YYYY format, if the date is not mentioned, answer Unknown, if the date is approximate, answer with a range of dates"
• "When was the breach discovered, be as accurate as you can"
• "Is the cause of the breach known - answer Yes or No only"
• "If the cause of the breach is known state it"
• "Were there any third parties involved - answer only Yes or No"
• "If there were third parties involved, list their names"

The idea is simple, give the SLM a number of press reports. Get it to answer the questions on each article. Check the accuracy of the results for each SLM.

As it turns out, my questions needs some work, but they’re good enough to get started.

Where to run your SLM?

The first choice you face is which computer to run your SLM on. Your choices boil down to evaluating it on the cloud or on your local machine. If you evaluate on the cloud, you need to choose a machine that’s powerful enough but also works with your budget. Of course, the advantage of cloud deployment is you can choose any machine you like. If you choose your local machine, it needs to be powerful enough for the job. The advantage of local deployment is that it’s easier and cheaper to get started.

To get going quickly, I chose my local machine, but as it turned out, it wasn’t quite powerful enough.

The code

This is where we part ways with the Ollama app and turn to coding.

The first step is installing the Ollama Python module (https://github.com/ollama/ollama-python). Unfortunately, the documentation isn’t great, so I’m going to help you through it.

We need to install the SLMs on our machine. This is easy to do, you can either do it via the command line or via the API. I’ll just show you the command line way to install the model llama3.2:

ollama pull llama3.2

Because we have the same nine questions we want to ask of each article, I’m going to create a ‘custom’ SLM. This means selecting a model (e.g. Llama3.2) and customizing it with my questions. Here’s my code.

ollama.create(

model=‘breach_analyzer’,

from_=‘llama3.2’,

system=system_prompt,

stream=True,

):

The system_prompt is my nine questions I showed you earlier plus a general prompt. model is the name I’m giving my custom model; in this case I’m calling it breach_analyzer.

Now I’ve customized my model, here’s how I call it:

response = ollama.generate(

model=‘breach_analyzer’,

prompt=prompt,

format=BreachAnalysisResponse.model_json_schema(),

)

The prompt is the text of the article I want to analyze. The format is the JSON format I want the results to be in. The response is the response from the model using the JSON format defined by BreachAnalysisResponse.model_json_schema().

Note I’m using generate here and not chat. My queries are "one-off" and there’s no sense of a continuing dialog. If I’d wanted a continuing dialog, I’d have used the chat function.

Here’s how my code works overall:

1. Read in the text from six online articles.
2. Load the model the user has selected (either mistral, llama3.2, or phi3).
3. Customize the model.
4. Run all six online articles through the customized model.
5. Collect the results and analyze them.

The results

The first thing I discovered is that these models are resource hogs! They hammered my machine and took 10-20 minutes to run each evaluation of six articles. My laptop is a 2020 developer spec MacBook Pro but it isn’t really powerful enough to evaluate SLMs. The first lesson is, you need a powerful, recent machine to make this work; one that has GPUs built in that the SML can access. I’ve heard from other people that running SLMs on high-spec machines leads to fast (usable) response times.

The second lesson is accuracy. Of the three models I evaluated, not all of them answered my questions correctly. One of the articles was an article about tennis and not about data breaches, but one of the models incorrectly said it was about data breaches. Another of the models told me it was unclear whether there were third parties involved in a breach and then told me the name of the third party!

On reflection, I needed to tweak my nine questions to get clearer answers. But this was difficult because of the length of time it took to analyze each article. This is a general problem; it took so long to run the models that any tweaking of code or settings took too much time.

The overall winner in terms of accuracy was Phi-3, but this was also the slowest to run on my machine, taking nearly 20 minutes to analyze six articles. From commentary I’ve seen elsewhere, this model runs acceptably fast on a more powerful machine.

Here’s the key question: could I replace paid-for LLMs with SLMs? My answer is: almost certainly yes, if you deploy your SLMs on a high-spec computer. There’s certainly enough accuracy here to warrant a serious investigation.

How I could have improved the results?

The most obvious thing is a faster machine. A brand new top-of-the-range MacBookPro with lots of memory and built-in GPUs. Santa, if you’re listening, this is what I’d like. Alternatively, I could have gone onto the cloud and used a GPU machine.

My prompts could be better. They need some tweaking.

I get the text of these articles using requests. As part of the process, it gives me all of the text on the page, which includes a lot of irrelevant stuff. A good next step would be to get rid of some of the extraneous and distracting text. There are lots of ways to do that and it’s a job any competent programmer could do.

If I could solve the speed problem, it would be good to investigate using multiple models. This could take several forms:

• asking the same questions using multiple models and voting on the results
• using different models for different questions.

What’s notable about these ways of improving the results is how simple they are.

Some musings

• Evaluating SLMs is firmly in the technical domain. I’ve heard of non-technical people try to play with these models, but they end up going nowhere because it takes technical skills to make them do anything useful.
• There are thousands of models and selecting the right one for your use case can be a challenge. I suggest going with the most recent and/or ones that score most highly on the HuggingFace leaderboard.
• It takes a powerful machine to run these models. A new high-end machine with GPUs would probably run these models "fast enough". If you have a very recent and powerful local machine, it’s worth playing around with SLMs locally to get started, but for serious evaluation, you need to get on the cloud and spend money.
• Some US businesses are allergic to models developed in certain countries, some European businesses want models developed in Europe. If the geographic origin of your model is important, you need to check before you start evaluating.
• You can get cost savings compared to LLMs, but there’s hard work to be done implementing SLMs.

I have a lot more to say about evaluations and SLMs that I’m not saying here. If you want to hear more, reach out to me.

Next steps

Ian Stokes-Rees gave an excellent tutorial at PyData Boston on this topic and that’s my number one choice for where to go next.

After that, I suggest you read the Ollama docs and join their Discord server. After that, the Hugging Face Community is a good place to go. Lastly, look at the YouTube tutorials out there.