Aarush Sah

Jul 31, 2025

Evaluating large language models (LLMs) today is fundamentally broken.

If you’ve spent any time with eval frameworks, you already know the drill: each one makes different decisions on how to prompt models, parse responses, and measure metrics like accuracy. Every lab has its own approach – pass@k, best-of-n, zero-shot, few-shot, CoT prompting… the list goes on. It’s subtle, but it means you can never truly compare numbers across different frameworks or model releases.

Even when everything is meticulously documented, reproducing results is frustratingly hard. Tiny implementation quirks creep in everywhere. In practice, benchmark scores are basically irreproducible.

We ran into this problem at Groq enough times to finally decide: okay, let’s fix this once and for all. We built OpenBench internally, and it genuinely helped us. So now we’re releasing it publicly because reliable evaluations shouldn’t just be our luxury, but a given for all.

Lack of Standardization

The core problem? Integration nightmares.

Here’s the landscape:

• Tons of frameworks (DeepEval, HELM, LM-Evaluation-Harness)
• Many benchmarks, each with their unique quirks and formats
• Metrics that aren’t even consistent across implementations
• Model APIs that behave completely differently from each other

As a result, evaluation teams get lost in what we’ve started calling "glue hell" - endless infrastructure, scripts, configs, debugging. It’s exhausting. Want to fairly compare models like o3, Claude Opus 4, and Qwen 3 on a bunch of different evals? Prepare for pain.

Common Pain Points

• Too Much Setup. Hours lost to configurations and troubleshooting.
• Metric Inconsistency. "Accuracy" and specific metrics don’t mean the same thing across frameworks.
• Wasted Compute. Sequential evals/frameworks ignore the inherent parallelism of evals (per-sample basis), making everything slow.
• Gaming Benchmarks. Custom setups artificially boost numbers without real-world meaning.

OpenBench: Benchmarks Done Right

OpenBench tackles these problems directly:

• Standardized Implementations. Every benchmark has exactly one canonical, robust implementation. MMLU is always MMLU, regardless of whether you’re testing o3, R1, Opus 4, or another model.
• Runs Everywhere. Built on a reliable integration layer powered by Inspect so evals run seamlessly across different models and APIs without fuss.

Here’s all you need to run your first evaluation (Prerequisite: Install uv):

1# Create a virtual environment and install OpenBench (30 seconds)
2uv venv
3source .venv/bin/activate
4uv pip install openbench
5# Set your API key (any provider!)
6export GROQ_API_KEY=your_key  # or OPENAI_API_KEY, ANTHROPIC_API_KEY, etc.
7# Run your first eval (30 seconds)
8bench eval mmlu --model groq/llama-3.3-70b-versatile --limit 10
9# That's it! 🎉 Check results in ./logs/ or view them in an interactive UI:
10bench view

Leveraging Inspect

We didn’t reinvent everything from scratch. OpenBench is built on Inspect from the UK AI Security Institute, which handles all the messy API details, rate limits, and response parsing complexities. This means we can focus purely on robust benchmark implementations. If you love OpenBench, please give them a star. We couldn’t have done it without them.

What’s Next? Community-Powered v0.1

OpenBench 0.1 launches with 18 carefully implemented benchmarks. But we’re far from done:

• New benchmarks every week for the next eight weeks – academic, coding, reasoning tasks, and more.
• Ongoing improvements driven by community feedback.
• Active collaboration – tell us what you need, submit new benchmarks, or suggest improvements.

We want OpenBench to solve your evaluation headaches, and your input is essential.

Try OpenBench

You can run your first eval right now - in under a minute (Prerequisite: Install uv):

1# Create a virtual environment and install OpenBench (30 seconds)
2uv venv
3source .venv/bin/activate
4uv pip install openbench
5# Set your API key (any provider!)
6export GROQ_API_KEY=your_key  # or OPENAI_API_KEY, ANTHROPIC_API_KEY, etc.
7# Run your first eval (30 seconds)
8bench eval mmlu --model groq/llama-3.3-70b-versatile --limit 10
9# That's it! 🎉 Check results in ./logs/ or view them in an interactive UI:
10bench view

**Join us here: **github.com/groq/openbench

**Learn more here: **openbench.dev/

Our objective: Robust evals, consistent results, no headaches.

Have feedback, feature requests, or just want to follow along as we build? Ping @groqinc or @AarushSah_ on X – our DMs are always open.