It’s an open secret that LLM benchmarks are bullshit. I built ReasonScape to be different, lets see what it tells us about how AI21’s latest drop compared to the high quality 4B we know and love.

My usual disclaimer is that these are all information processing tasks so I make no claims of performance on summarization, creative writing or similar tasks. This evaluation is a counting letters, tracking objects, doing math, following instructions kinda thing.

The second disclaimer is that I am sharing data from my development branch that’s not yet been published to the leaderboard or explorer apps - working on it, aiming for this weekend.

Caveats aside lets start with high-level views:

Overview

In terms of average tokens, this model sits somewhere between the OG and 2507-Thinking. Performance was incredibly weak outside of 2 domains: Cars (Spatial state tracking) and Dates (Time operations).

The reasonscape methodology requires me to run *a lot* of tests, but also gives us a way to look deeper inside the performance of each task:

Task Deep Dive 1: Arithmetic, Boolean, Brackets, Cars, Shuffle, Objects Task Deep Dive 2: Dates, Letters, Movie, Sequence, Shapes, Sort

The original Qwen3-4B was a really strong model, the 2507 release that split it into two halves was a mixed bag - the resulting Thinking model is quite good but it does not universally outperform the OG, Sequence is an example of a task the 2507 regressed on.

Before final thoughts, lets directly compare Jamba to the OG Qwen3-4B across the gamut of tasks:

Bar Plot: Jamba Reasoning 3B Bar Plot: Qwen3-4B OG

I think it’s fair to say that task-performance of Jamba Reasoning 3B leaves much to be desired. Letters is a parametric version the ‘count rs in strawberry’ tests and for a native-thinking model to fail it this hard is pretty embarassing imo.

The glaring problem with this model is truncation. All these evaluations were run at 8K context, so naively you’d think truncation means the model is just running out of tokens but this is absolutely not the case - if you look at Dates for example all successfull responces are ~2K but truncation rate is still a crazy ~10% the just model loses its mind:

We need to parse the problem: "Given a small set of sentences about a particular date, answer the provided question. Respond only with the final date in MM/DD/DD/YYYY[*six thousand more Y*]

We need to find today's date given that Xavier moved to the city on 04/11/2023, and 299 days have passed since then. So we need to add 299 days to 04/11/2023.\n\nWe need to compute the date.\n\nLet's do step by step.\n\nStart date: April 11, 2023.\n\nAdd 299 days.\n\nWe can break into years, months, days.\n\nFirst, add full years. Since 299 days is less than a year, we can approximate.\n\nBut we need to be precise.\n\nWe can use a date calculator approach.\n\nLet's compute:\n\nApril 11, 2023 + 299 days.\n\nWe can add month by month, but need to account for days.\n\nLet's do it systematically.\n\nWe can use a date addition: 299 days from April 11, 2023.\n\nWe can break into quarters maybe.\n\nBut maybe easier: compute month by month.\n\nWe can start from April 11, 2023.\n\nAdd 30 days to get May 11, 2023.\n\nAdd another 30 days: June 11, 2023.\n\nAdd another 30 days: July 11, 2023.\n\nAdd another 30 days: August 11, 2023.\n\nAdd another 30 days: September 11, 2023.\n\nAdd another 30 days: October 11, 2023.\n\nAdd another 30 days: November 11, 2023.\n\nWait, but we only have 299 days. Let's see.\n\nWe can count the days month by month.\n\nFrom April 11 to next April 11 is 365 days? But we only add 299 days.\n\nLet's do a more precise method.\n\nWe can use the fact that 299 days from April 11, 2023.\n\nWe can compute by adding days month by month.\n\nApril 11, 2023 + 299 days.\n\nWe can break into months:\n\n- April: 30 days (from April 11 to April 30). That's 19 days? Wait, from April 11 to May 11 is 30 days? Let's do step by step.\n\n\n\n\nAlternatively, we can use a date calculator logic.\n\nLet's use a simpler approach: add 299 days to April 11, 2023.\n\nWe can use the fact that 2023 is not a leap year (2023 is not divisible by 4). So February has 28 days.\n\nWe can use a date addition.\n\nBut maybe we can use a known method: add 299 days to April 11.\n\nWe can add 365 days to get next year, but we only need 299.\n\nLet's do step by step:\n\nApril 11, 2023.\n\nAdd 1 day: April 12, 2023.\n\nAdd 298 days: we need to add 298 days.\n\nWe can break 298 days into years, months, days.\n\nBut maybe we can use a simpler approach: find the date by adding 299 days to April 11, 2023.\n\nWe can use a date calculator mental.\n\nLet's add 299 days to April 11, 2023.\n\nApril 11 + 299 days.\n\nWe can compute:\n\n- April has 30 days, so adding 30 days would get to May 0? Wait.\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nThis is too messy.\n\nLet'solve the date calculation.\nWe need to compute the current date:\nThe task: Given a small set of sentences about a particular date.\n\nThe task:\nPlease write a function that is to\n<think>\nWe need to write a function that returns the\n\nWe need to\n</think>\n\nWe need to\n\n\n\n<think>\n</think>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n</think>\nWe need to write a small set of the problem.\n\n\n<think>\nWe need to write a function.\n\nWe need to think\n\n<think>\n\n\n\n\n\nWe need to think\n\n\n</think>\n</think>\n\n\n<think>\nWe need to start a new task:\n\n\nWe need to start a new task.\n\n\n<think>\n\nWe need to\n\nWe need to start a new start\n\n\n\n\n</think>\n\nWe need to start new start\n\n\n\n\n\n\n<think>\n\nWe need to start a new start\n\n\n\n\n\n\n</think>\n\nWe need to start start to start start start [start repeats 4000 times]

I ran all models with {“temperature”: 0.6, “top_p”: 0.95, “top_k”: 20, “min_p”: 0 } which is my standard sampler for reasoning models, perhaps there is a different configuration that works better for Jamba reasoning specifically.

In closing, I don’t believe this model is comparable to Qwen3-4B on practical tasks. It’s far worse at basically all tasks, and has a universal truncation problem.

Thanks for reading and keep it local! <3