Hatice Ozen

May 16, 2025

Simplifying the Complexity of AI Agents with Server-Side Tool Use

Large Language Models (LLMs) are powerful but constrained by static training data, lacking the ability to access real-time information or interact dynamically with external environments. We need real-time data, not snapshots from 2023. This limitation has driven the adoption of tool use (also known as function calling), which equips LLMs with tools or functions to fetch live data, execute code, and navigate complex tasks (including playing Pokémon).

This evolution has led us to AI agents, which are systems that leverage LLMs and tools to autonomously interact with external environments. Neon recently reported that over 80% of their databases were created by AI agents, outpacing human contributions by 4x. There are even AI agent marketplaces now where we can list and recruit agents for jobs.

Building AI agents, however, introduces significant complexity to orchestrate tool pipelines and ensure reliable results. If you’ve built AI agents before, you’re familiar with the challenges of having to spend more time debugging tool call failures than building features. Further, agentic workflows – where multiple AI agents autonomously orchestrate multiple tool calls in the backend – demand more than capability. They require speed. Latency in tool execution can bottleneck real-time applications, degrade user experience, and limit scalability, especially when agents iterate through web searches, computations, or various API integrations.

Image source: ChatGPT (generating this took forever - can’t wait until we get image generation models on Groq!)

Enter compound-beta** **and compound-beta-mini, built by Ben Klieger (give him a follow!) and available in preview on Groq API. These systems leverage server-side tool orchestration to abstract tool use complexity, enabling us to build powerful, real-time AI agents with minimal configuration.

compound-beta is a compound AI system that integrates multiple open-source models to handle complex, multi-step tasks. It supports simultaneous tool calls, such as web searches and Python code execution, making it ideal for iterative workflows. compound-beta-mini, a lightweight variant, focuses on single-tool tasks, prioritizing low latency for real-time applications.

By combining open-source models (Llama 4 Scout and Llama 3.3 70B), compound-beta delivers robust, multi-tool workflows, while compound-beta-mini optimizes for single-tool tasks with even lower latency. Both models are accessible via Groq API by specifying compound-beta or compound-beta-mini as the model parameter within a standard chat completion request, requiring no additional code.

Technical Specifications

Inference Speed

I could just say that compound-beta achieves ~350 tokens/second while compound-beta-mini achieves ~275 tokens/second, but this doesn’t illustrate how much time and effort these systems save us. As an example, I ran the following experiment on OpenAI’s gpt-4o with web search enabled, compound-beta-mini, OpenAI’s 3o, and compound-beta:

Query: Write an academic report about what Groq was in the news for this past week focusing on impact and implications.

Model	Number of Sources Used	Time Taken for Results(s)	Output Tokens (Length of Generated Report)
gpt-4o	3	4.26	312
compound-beta-mini	17	3.14	756
o3	Cited 6 sources ranging from 2023-2025 (not current)	54.04	1,926
compound-beta	17	10.54	2,419

Each tool call, whether fetching live news or executing code, adds latency. Groq hardware ensures these operations are near-instantaneous so we don’t have to get bored watching loading icons.

As shown above (and you can experiment yourself on Groq Console Playground), compound systems on Groq deliver dramatically more comprehensive research! While gpt-4o accessed 3 sources from web search, compound-beta-mini leveraged 17 different sources – that’s nearly 6x more research depth – while running faster than gpt-4o and delivering a more comprehensive, detailed report that was twice as long. Compared to o3, compound-beta outperforms by searching 3x more sources (all current), generating a more detailed analysis, and delivering results 5x faster. As a side note, in my experimentation, I kept getting hallucinated and outdated sources from o3, which was interesting.

Let’s think of compound-beta-mini’s results of a report with information cited from 17 sources produced in 3.14 second. For us humans, that effectively means compressing a full day’s research work into the time it takes to read this sentence. This represents not just minutes saved, but a fundamental transformation in how research can be conducted.

Quality

Speed is crucial for AI agents, but so is quality. Thanks to a lack of evaluations for live data fetching, we open-sourced the RealtimeEval benchmark that can be automatically updated with information from the past 24 hours designed to evaluate tool use for current events. On this benchmark, compound-beta outperformed GPT-4o-search-preview and GPT-4o-mini-search-preview significantly with performance on par with Perplexity Sonar.

Ease of Integration

If you’re migrating from existing agent frameworks, both compound-beta and compound-beta-mini integrate naturally with your current stack – all you need to do is set the model parameter in your standard Groq API call to one of them.

If you’re building from scratch, start with the tutorial below and see more code examples in our documentation.

Building a Foreign Policy Research AI Agent (Python)

Now that we’ve learned about how powerful compound-beta and compound-beta-mini are, let’s see how simple it is to build a research agent that fetches today’s foreign policy news involving President Trump, restricts sources to cnn.com and foxnews.com domains, and generates a tweet summarizing key updates using compound-beta’s fast real-time web search and reasoning via a single Groq API call.

Step 1: Environment Setup

Obtain a free Groq API key from Groq Console and install the Python SDK (we have a generous free tier with thousands of tokens per minute for you to play with across all available models):

1pip install groq

Step 2: Initialize the Client

Set your Groq API key as an environment variable and initialize the Groq Client:

import os from groq import Groq client = Groq(api_key=os.environ.get("GROQ_API_KEY"))

1import os
2from groq import Groq
3
4client = Groq(api_key=os.environ.get("GROQ_API_KEY"))

Step 3: Build Your Research Agent

This code configures compound-beta for invoking web search tools to analyze recent developments in foreign policy and to generate a tweet:

1import os
2from groq import Groq
3
4client = Groq(api_key=os.environ.get("GROQ_API_KEY"))
5
6def foreign_policy_research_agent(query):
7
8   system_prompt = (
9       "You are a foreign policy research assistant tasked with delivering a rigorous analysis. "
10       "CRITICAL INSTRUCTION: You must ONLY use news sources published on the EXACT date specified in the user's query. "
11       "Follow these strict guidelines for date filtering:"
12       "\n\n1. First, extract and identify the specific target date from the user's query (format: Month Day, Year)"
13       "\n2. Parse any search results to verify their publication dates"
14       "\n3. COMPLETELY DISCARD any information from sources not published on the exact target date"
15       "\n4. Begin your response by stating: 'Analysis based EXCLUSIVELY on news published on [TARGET DATE]:'"
16       "\n5. For each piece of information, cite the source and confirm its publication date"
17       "\n6. If you cannot find reliable sources from the exact date requested, clearly state: 'No reliable sources found from [TARGET DATE] on this topic.'"
18       "\n\nFor the query, provide: "
19       "- Core foreign policy concepts and context that involves President Donald Trump\n"
20       "- Critical analysis of implications and trends\n"
21       "Also generate a tweet (≤280 characters) with specific key updates. Do not use hashtags."
22       "\n\nNEVER use information from sources published before or after the target date, even if the information seems relevant."
23   )
24
25   response = client.chat.completions.create(
26       model="compound-beta",
27       messages=[
28           {"role": "system", "content": system_prompt},
29           {"role": "user", "content": query},
30       ],
31       include_domains=["cnn.com", "foxnews.com"],
32       max_tokens=2000,
33       temperature=0.6,
34       top_p=0.95,
35       stream=False,
36   )
37
38   return response.choices[0].message.content, response.choices[0].message.executed_tools
39
40# Execute the query with the date embedded in the query
41query = "Foreign policy news involving President Donald Trump on May 13, 2025"
42result, tools_used = foreign_policy_research_agent(query)
43print("Analysis:\n", result)
44print("Tools Used:\n", tools_used)

Running the above code results in the following output from compound-beta (with “Tools Used” condensed for brevity):

1Analysis:
2 Analysis based EXCLUSIVELY on news published on May 13, 2025:
3
4On May 13, 2025, President Donald Trump's foreign policy efforts were prominently featured in various news sources. According to CNN, Trump's visit to the Middle East, particularly his trip to Saudi Arabia, was a significant development (CNN, May 13, 2025). During his visit, the Trump administration signed several new agreements with Saudi Arabia, including on military cooperation, as reported by CNN (CNN, May 13, 2025). This move highlights the close ties between the US and Saudi Arabia, with the latter seeking to showcase its capabilities as the Middle East undergoes rapid changes.
5
6Additionally, Fox News reported that some Biden officials have praised Trump's bold Middle East foreign policy moves (Fox News, May 13, 2025). Trump's visit to Saudi Arabia and the agreements signed during the trip are seen as significant developments in the region.
7
8In terms of implications and trends, Trump's foreign policy efforts appear to be focused on strengthening alliances and mitigating crises caused by his administration's previous actions. The agreements with Saudi Arabia and potential progress in US-China trade negotiations demonstrate Trump's attempts to assert his influence in global affairs, as reported by CNN (CNN, May 13, 2025).
9
10However, it is essential to note that these developments may also be perceived as attempts to consolidate power and distract from domestic issues. The ongoing tariff war with China and tensions in the Middle East remain critical areas to monitor.
11
12Some key points to consider:
13
14* Trump's visit to Saudi Arabia and the signing of military cooperation agreements (CNN, May 13, 2025)
15* Biden officials' praise for Trump's Middle East foreign policy moves (Fox News, May 13, 2025)
16* Potential progress in US-China trade negotiations (CNN, May 13, 2025)
17* Ongoing tariff war with China and tensions in the Middle East (CNN, May 13, 2025)
18
19Tweet:
20"President Donald Trump signs military cooperation agreements with Saudi Arabia during his visit to Riyadh, highlighting close US-Saudi ties. Trump's foreign policy efforts focus on strengthening alliances and mitigating crises."
21
22Tools Used:
23 [ExecutedTool(arguments='{"query": "Foreign policy news involving President Donald Trump on May 13, 2025"}', index=0, type='search', output='Title: May 13, 2025 - US President Donald Trump\'s Middle East trip - CNN\nURL: https://www.cnn.com/politics/live-news/trump-middle-east-news-05-13-25\nContent: May 13, 2025 - US President Donald Trump's Middle East trip | CNN Politics • Trump in Riyadh:The Trump administration signed several new agreements with Saudi Arabia today, including on military cooperation, as President Donald Trump and Crown Prince Mohammed bin Salman sought to highlight the countries' close ties. For Saudi Arabia, US President Donald Trump's visit is an opportunity for the kingdom to showcase what it has to offer as the Middle East rapidly changes, a regional expert said. Tech billionaire Elon Musk will be in Saudi Arabia during President Donald Trump's visit to the kingdom, a White House official has told CNN.\nScore: 0.8374\n\nTitle: Biden officials impressed by Trump\'s bold Middle East foreign policy ...\nURL: https://www.foxnews.com/media/former-biden-officials-offer-rare-praise-trumps-bold-middle-east-moves\nContent: U.S. President Donald J. Trump addresses the audience at the King Abdul Aziz International Conference Center while attending a Saudi-U.S. business investment forum, on May 13, 2025, in Riyadh\nScore: 0.6922\n)]

Both compound-beta and compound-beta-mini have a field called executed_tools to view the tools and related information used, which is great for citing sources and debugging.

So, What Next?

Speed, simplicity, and reliability are necessities, not luxuries, in agentic workflows. Each backend tool call, such as searching the news like our above tutorial, must successfully execute rapidly to meet real-time demands. Thankfully, we now have compound-beta and compound-beta-mini setting the stage for scalable, speed-driven agentic systems.

Here at Groq, we love our developer community and are excited to get your feedback on other tools that we should add to our compound AI systems. Have you tried compound-beta and compound-beta-mini yet? Let me know your experiences on X or in our Discord Community.

As always, happy building!