usually starts the same way. In a leadership meeting, someone says: “Let’s use AI!” Heads nod, enthusiasm builds, and before you know it, the room lands on the default conclusion: “Sure — we’ll build a chatbot.” That instinct is understandable. Large language models are powerful, ubiquitous, and fascinating. They promise intuitive access to universal knowledge and functionality.

The team walks away and starts building. Soon, demo time comes around. A polished chat interface appears, accompanied by confident arguments about why this time, it will be different. At that point, however, it usually hasn’t reached real users in real situations, and evaluation is biased and optimistic. Someone in the audience inevitably comes up with a custom question, irritating the bot. The developers promise to fix “it”, but in most cases, the underlying issue is systemic.

Once the chatbot hits the ground, initial optimism is often matched by user frustration. Here, things get a bit personal because over the past weeks, I was forced to spend some time talking to different chatbots. I tend to delay interactions with service providers until the situation becomes unsustainable, and a couple of these cases had piled up. Smiling chatbot widgets became my last hope before an eternal hotline call, but:

• After logging in to my car insurer’s site, I asked to explain an unannounced price increase, only to realize the chatbot had no access to my pricing data. All it could offer was the hotline number. Ouch.
• After a flight was canceled at the last minute, I asked the airline’s chatbot for the reason. It politely apologized that, since the departure time was already in the past, it couldn’t help me. It was open to discuss all other topics, though.
• On a telco site, I asked why my mobile plan had suddenly expired. The chatbot confidently replied that it couldn’t comment on contractual matters and referred me to the FAQs. As expected, these were long but irrelevant.

These interactions didn’t bring me closer to a solution and left me at the opposite end of delight. The chatbots felt like foreign bodies. Sitting there, they consumed real estate, latency, and attention, but didn’t add value.

Let’s skip the debate on whether these are intentional dark patterns. The fact is, legacy systems as the above carry a heavy burden of entropy. They come with tons of unique data, knowledge, and context. The moment you try to integrate them with a general-purpose LLM, you make two worlds clash. The model needs to ingest the context of your product so it can reason meaningfully about your domain. Proper context engineering requires skill and time for relentless evaluation and iteration. And before you even get to that point, your data needs to be ready, but in most organizations, data is noisy, fragmented, or just missing.

In this post, I’ll recap insights from my book The Art of AI Product Development and my recent talk at the Google Web AI Summit and share a more organic, incremental approach to integrating AI into existing products.

Using smaller models for low-risk, incremental AI integration

“When implementing AI, I see more organizations fail by starting too big than starting too small.” ( Andrew Ng).

AI integration needs time:

• Your technical team needs to prepare the data and learn the available techniques and tools.
• You need to prototype and iterate to find the sweet spots of AI value in your product and market.
• Users need to calibrate their trust when moving to new probabilistic experiences.

To adapt to these learning curves, you shouldn’t rush to expose AI — especially open-ended chat functionality — to your users. AI introduces uncertainty and mistakes into the experience, which most people don’t like.

One effective way to pace your AI journey in the brownfield context is by using small language models (SLMs), which typically range from a few hundred million to a few billion parameters. They can integrate flexibly with your product’s existing data and infrastructure, rather than adding more technological overhead.

How SLMs are trained

Most SLMs are derived from larger models through knowledge distillation. In this setup, a large model acts as the teacher and a smaller one as the student. For example, Google’s Gemini served as the teacher for Gemma 2 and Gemma 3 , while Meta’s Llama Behemoth trained its herd of smaller Llama 4 models. Just as a human teacher condenses years of study into clear explanations and structured lessons, the large model distills its vast parameter space into a smaller, denser representation that the student can absorb. The result is a compact model that retains much of the teacher’s competence but operates with far fewer parameters and dramatically lower computational cost.

Figure 1: In knowledge distillation, the smaller student model soaks in the essential knowledge of a larger teacher model

Using SLMs

One of the key advantages of SLMs is their deployment flexibility. Unlike LLMs that are mostly used through external APIs, smaller models can be run locally, either on your organization’s infrastructure or directly on the user’s device:

• Local deployment: You can host SLMs on your own servers or within your cloud environment, keeping full control over data, latency, and compliance. This setup is ideal for enterprise applications where sensitive information or regulatory constraints make third-party APIs impractical.

📈 Local deployment also offers you flexible fine-tuning opportunities as you collect more data and need to respond to growing user expectations.

• On-device deployment via the browser: Modern browsers have built-in AI capabilities that you can rely on. For instance, Chrome integrates Gemini Nano via the built-in AI APIs, while Microsoft Edge includes Phi-4 (see Prompt API documentation). Running models directly in the browser enables low-latency, privacy-preserving use cases such as smart text suggestions, form autofill, or contextual help.

If you would like to learn more about the technicalities of SLMs, here are a couple of useful resources:

• Overview of SLMs and other types of language models: Lipenkova, Janna (2025). The Art of AI Product Development, Chapter 5: “Exploring and Evaluating Language Models.”
• Comparison of modern SLMs: Lu, Zhenyan et al (2025). Small Language Models: Survey, Measurements, and Insights. Preprint available at https://arxiv.org/pdf/2409.15790 arXiv
• SLM deployment and fine-tuning: Iozzia, Guglielmo (2025). Domain-Specific Small Language Models.

Let’s now move on and see what you can build with SLMs to provide user value and make steady progress in your AI integration.

Product opportunities for SLMs

SLMs shine in focused, well-defined tasks where the context and data are already known — the kinds of use cases that live deep inside existing products. You can think of them as specialized, embedded intelligence rather than general-purpose assistants. Let’s walk through the main buckets of opportunity they unlock in the brownfield, as illustrated in the following opportunity tree.

Figure 2: Opportunity categories for SLM, with examples from the travel industry

1. Better product analytics

Before exposing AI features to users, look for ways to improve your product from the inside. Most products already generate a continuous stream of unstructured text — support chats, help requests, in-app feedback. SLMs can analyze this data in real time and surface insights that inform both product decisions and immediate user experience. Here are some examples:

• Tag and route support chats as they happen, directing technical issues to the right teams.
• Flag churn signals during a session, prompting timely interventions.
• Suggest relevant content or actions based on the user’s current context.
• Detect repeated friction points while the user is still in the flow, not weeks later in a retrospective.

These internal enablers keep risk low while adding value and giving your team time to learn. They strengthen your data foundation and prepare you for more visible, user-facing AI features down the road.

2. Remove friction

Next, take a step back and audit UX debt that’s already there. In the brownfield, most products aren’t exactly a designer’s dream. They were designed under the technical and architectural constraints of their time. With AI, we now have an opportunity to lift some of those constraints, reducing friction and creating faster, more intuitive experiences.

A good example is the smart filters on search-based websites like Booking.com. Traditionally, these pages use long lists of checkboxes and categories that try to cover every possible user preference. They’re cumbersome to design and use, and in the end, many users can’t find the setting that matters to them.

Language-based filtering changes this. Instead of navigating a complex taxonomy, users simply type what they want (for example “pet-friendly hotels near the beach”), and the model translates it into a structured query behind the scenes.

Figure 3: Smart, language-driven filters remove the cognitive load of long filter bars

More broadly, look for areas in your product where users need to apply your internal logic — your categories, structures, or terminology — and replace that with natural language interaction. Whenever users can express intent directly, you remove a layer of cognitive friction and make the product smarter and friendlier.

3. Augment

With your user experience decluttered, it’s time to think about augmentation — adding small, useful AI capabilities to your product. Instead of reinventing the core experience, look at what users are already doing around your product — the side tasks, workarounds, or external tools they rely on to reach their goal. Can focused AI models help them do it faster or smarter?

For example, a travel app could integrate a contextual trip note generator that summarizes itinerary details or drafts messages for co-travelers. A productivity tool could include a meeting recap generator that summarizes discussions or action items from text notes, without sending data to the cloud.

These features grow organically from real user behavior and extend your product’s context instead of redefining it.

4. Personalize

Successful personalization is the holy grail of AI. It flips the traditional dynamic: instead of asking users to learn and adapt to your product, your product now adapts to them like a well-fitting glove.

When you start, keep ambition at bay — you don’t need a fully adaptive assistant. Rather, introduce small, low-risk adjustments in what users see, how information is phrased, or which options appear first. At the content level, AI can adapt tone and style, like using concise wording for experts and more explanatory phrasing for newcomers. At the experience level, it can create adaptive interfaces. For instance, a project-management tool could surface the most relevant actions (“create task,” “share update,” “generate summary”) based on the user’s past workflows.

⚠️ When personalization goes wrong, it quickly erodes trust. Users sense that they’ve traded personal data for an experience that doesn’t feel better. Thus, introduce personalization only once your data is ready to support it.

Why “small” wins over time

Each successful AI feature — be it an analytics improvement, a frictionless UX touchpoint, or a personalized step in a larger flow — strengthens your data foundation and builds your team’s iteration muscle and AI literacy. It also lays the groundwork for larger, more complex applications later. When your “small” features work reliably, they become reusable components in bigger workflows or modular agent systems (cf. Nvidia’s paper Small Language Models are the Future of Agentic AI).

To summarize:

✅ Start small — favor gradual improvement over disruption.

✅ Experiment fast — smaller models mean lower cost and faster feedback loops.

✅ Be cautious — start internally; introduce user-facing AI once you’ve validated it.

✅ Build your iteration muscle — steady, compound progress beats headline projects.

Originally published at https://jannalipenkova.substack.com.