Abstract

Hierarchical Reasoning Model (HRM) is a novel approach using two small neural networks recursing at different frequencies. This biologically inspired method beats Large Language models (LLMs) on hard puzzle tasks such as Sudoku, Maze, and ARC-AGI while trained with small models (27M parameters) on small data (∼\sim 1000 examples). HRM holds great promise for solving hard problems with small networks, but it is not yet well understood and may be suboptimal. We propose Tiny Recursive Model (TRM), a much simpler recursive reasoning approach that achieves significantly higher generalization than HRM, while using a single tiny network with only 2 layers. With only 7M parameters, TRM obtains 45% test-accuracy on ARC-AGI-1 and 8% on ARC-AGI-2, higher than most LLMs (e.g., Deepseek R1, o3-mini, Gemini 2.5 Pro) with less than 0.01% of the parameters.

reasoning, recurrent, arc-agi

Alexia Jolicoeur-Martineau

Samsung SAIL Montréal

alexia.j@samsung.com

1 Introduction

While powerful, Large Language models (LLMs) can struggle on hard question-answer problems. Given that they generate their answer auto-regressively, there is a high risk of error since a single incorrect token can render an answer invalid. To improve their reliability, LLMs rely on Chain-of-thoughts (CoT) (Wei et al., 2022) and Test-Time Compute (TTC) (Snell et al., 2024). CoTs seek to emulate human reasoning by having the LLM to sample step-by-step reasoning traces prior to giving their answer. Doing so can improve accuracy, but CoT is expensive, requires high-quality reasoning data (which may not be available), and can be brittle since the generated reasoning may be wrong. To further improve reliability, test-time compute can be used by reporting the most common answer out of KK or the highest-reward answer (Snell et al., 2024).

Figure 1: Tiny Recursion Model (TRM) recursively improves its predicted answer yy with a tiny network. It starts with the embedded input question xx and initial embedded answer yy, and latent zz. For up to Ns​u​p=16N_{sup}=16 improvements steps, it tries to improve its answer yy. It does so by i) recursively updating nn times its latent zz given the question xx, current answer yy, and current latent zz (recursive reasoning), and then ii) updating its answer yy given the current answer yy and current latent zz. This recursive process allows the model to progressively improve its answer (potentially addressing any errors from its previous answer) in an extremely parameter-efficient manner while minimizing overfitting.

However, this may not be enough. LLMs with CoT and TTC are not enough to beat every problem. While LLMs have made significant progress on ARC-AGI (Chollet, 2019) since 2019, human-level accuracy still has not been reached (6 years later, as of writing of this paper). Furthermore, LLMs struggle on the newer ARC-AGI-2 (e.g., Gemini 2.5 Pro only obtains 4.9% test accuracy with a high amount of TTC) (Chollet et al., 2025; ARC Prize Foundation, 2025b).

An alternative direction has recently been proposed by Wang et al. (2025). They propose a new way forward through their novel Hierarchical Reasoning Model (HRM), which obtains high accuracy on puzzle tasks where LLMs struggle to make a dent (e.g., Sudoku solving, Maze pathfinding, and ARC-AGI). HRM is a supervised learning model with two main novelties: 1) recursive hierarchical reasoning, and 2) deep supervision.

Recursive hierarchical reasoning consists of recursing multiple times through two small networks (fLf_{L} at high frequency and fHf_{H} at low frequency) to predict the answer. Each network generates a different latent feature: fLf_{L} outputs zHz_{H} and fHf_{H} outputs zLz_{L}. Both features (zL,zHz_{L},z_{H}) are used as input to the two networks. The authors provide some biological arguments in favor of recursing at different hierarchies based on the different temporal frequencies at which the brains operate and hierarchical processing of sensory inputs.

Deep supervision consists of improving the answer through multiple supervision steps while carrying the two latent features as initialization for the improvement steps (after detaching them from the computational graph so that their gradients do not propagate). This provide residual connections, which emulates very deep neural networks that are too memory expensive to apply in one forward pass.

An independent analysis on the ARC-AGI benchmark showed that deep supervision seems to be the primary driver of the performance gains (ARC Prize Foundation, 2025a). Using deep supervision doubled accuracy over single-step supervision (going from 19%19% to 39%39% accuracy), while recursive hierarchical reasoning only slightly improved accuracy over a regular model with a single forward pass (going from 35.7%35.7% to 39.0%39.0% accuracy). This suggests that reasoning across different supervision steps is worth it, but the recursion done in each supervision step is not particularly important.

In this work, we show that the benefit from recursive reasoning can be massively improved, making it much more than incremental. We propose Tiny Recursive Model (TRM), an improved and simplified approach using a much smaller tiny network with only 2 layers that achieves significantly higher generalization than HRM on a variety of problems. In doing so, we improve the state-of-the-art test accuracy on Sudoku-Extreme from 55% to 87%, Maze-Hard from 75% to 85%, ARC-AGI-1 from 40% to 45%, and ARC-AGI-2 from 5% to 8%.

2 Background

HRM is described in Algorithm 2. We discuss the details of the algorithm further below.

2.1 Structure and goal

The focus of HRM is supervised learning. Given an input, produce an output. Both input and output are assumed to have shape [B,L][B,L] (when the shape differs, padding tokens can be added), where BB is the batch-size and LL is the context-length.

HRM contains four learnable components: the input embedding fI​(⋅;θI)f_{I}(\cdot;\theta_{I}), low-level recurrent network fL​(⋅;θL)f_{L}(\cdot;\theta_{L}), high-level recurrent network fH​(⋅;θH)f_{H}(\cdot;\theta_{H}), and the output head fO​(⋅;θO)f_{O}(\cdot;\theta_{O}). Once the input is embedded, the shape becomes [B,L,D][B,L,D] where DD is the embedding size. Each network is a 4-layer Transformers architecture (Vaswani et al., 2017), with RMSNorm (Zhang & Sennrich, 2019), no bias (Chowdhery et al., 2023), rotary embeddings (Su et al., 2024), and SwiGLU activation function (Hendrycks & Gimpel, 2016; Shazeer, 2020).

def hrm(z, x, n=2, T=2):

zH, zL = z

with torch.no_grad():

for i in range(nT - 2):

zL = L_net(zL, zH, x)

if (i + 1) % T == 0:

zH = H_net(zH, zL)

zL = L_net(zL, zH, x)

zH = H_net(zH, zL)

return (zH, zL), output_head(zH), Q_head(zH)

def ACT_halt(q, y_hat, y_true):

target_halt = (y_hat == y_true)

loss = 0.5*binary_cross_entropy(q[0], target_halt)

return loss

def ACT_continue(q, last_step):

if last_step:

target_continue = sigmoid(q[0])

else:

target_continue = sigmoid(max(q[0], q[1])))

loss = 0.5*binary_cross_entropy(q[1], target_continue)

return loss

for x_input, y_true in train_dataloader:

z = z_init

for step in range(N_sup):

x = input_embedding(x_input)

z, y_pred, q = hrm(z, x)

loss = softmax_cross_entropy(y_pred, y_true)

loss += ACT_halt(q, y_pred, y_true)

_, _, q_next = hrm(z, x)

loss += ACT_continue(q_next, step == N_sup - 1)

z = z.detach()

loss.backward()

opt.step()

opt.zero_grad()

if q[0] > q[1]:

break

Figure 2: Pseudocode of Hierarchical Reasoning Models (HRMs).

2.2 Recursion at two different frequencies

Given the hyperparameters used by Wang et al. (2025) (n=2n=2 fLf_{L} steps, 1 fHf_{H} steps; done T=2T=2 times), a forward pass of HRM is done as follows:

	x\displaystyle x	←fI​(x~)\displaystyle\leftarrow f_{I}(\tilde{x})
	zL\displaystyle z_{L}	←fL​(zL+zH+x)​# without gradients\displaystyle\leftarrow f_{L}\left(z_{L}+z_{H}+x\right)\hskip 6.0pt\text{# without gradients}
	zL\displaystyle z_{L}	←fL​(zL+zH+x)​# without gradients\displaystyle\leftarrow f_{L}\left(z_{L}+z_{H}+x\right)\hskip 6.0pt\text{# without gradients}
	zH\displaystyle z_{H}	←fH​(zL+zH)# without gradients\displaystyle\leftarrow f_{H}\left(z_{L}+z_{H}\right)\hskip 22.0pt\text{# without gradients}
	zL\displaystyle z_{L}	←fL​(zL+zH+x)​# without gradients\displaystyle\leftarrow f_{L}\left(z_{L}+z_{H}+x\right)\hskip 6.0pt\text{# without gradients}
	zL\displaystyle z_{L}	←zL.d​e​t​a​c​h​()\displaystyle\leftarrow z_{L}.detach()
	zH\displaystyle z_{H}	←zH.d​e​t​a​c​h​()\displaystyle\leftarrow z_{H}.detach()
	zL\displaystyle z_{L}	←fL​(zL+zH+x)​# with gradients\displaystyle\leftarrow f_{L}\left(z_{L}+z_{H}+x\right)\hskip 6.0pt\text{# with gradients}
	zH\displaystyle z_{H}	←fH​(zL+zH)# with gradients\displaystyle\leftarrow f_{H}\left(z_{L}+z_{H}\right)\hskip 22.0pt\text{# with gradients}
	y^\displaystyle\hat{y}	←argmax​(fO​(zH))\displaystyle\leftarrow\text{argmax}(f_{O}\left(z_{H}\right))

where y^\hat{y} is the predicted output answer, zLz_{L} and zHz_{H} are either initialized embeddings or the embeddings of the previous deep supervision step (after detaching them from the computational graph). As can be seen, a forward pass of HRM consists of applying 6 function evaluations, where the first 4 function evaluations are detached from the computational graph and are not back-propagated through. The authors uses n=2n=2 with T=2T=2 in all experiments, but HRM can be generalized by allowing for an arbitrary number of L steps (nn) and recursions (TT) as shown in Algorithm 2.

2.3 Fixed-point recursion with 1-step gradient approximation

Assuming that (zLz_{L}, zHz_{H}) reaches a fixed-point (zL∗z_{L}^{{*}, zH∗z_{H}}{*}) through recursing from both fLf_{L} and fHf_{H},

	zL∗\displaystyle z_{L}^{*}	≈fL​(zL∗+zH+x)\displaystyle\approx f_{L}\left(z_{L}^{*}+z_{H}+x\right)
	zH∗\displaystyle z_{H}^{*}	≈fH​(zL+zH∗),\displaystyle\approx f_{H}\left(z_{L}+z_{H}^{*}\right),

the Implicit Function Theorem (Krantz & Parks, 2002) with the 1-step gradient approximation (Bai et al., 2019) is used to approximate the gradient by back-propagating only the last fLf_{L} and fHf_{H} steps. This theorem is used to justify only tracking the gradients of the last two steps (out of 6), which greatly reduces memory demands.

2.4 Deep supervision

To improve effective depth, deep supervision is used. This consists of reusing the previous latent features (zHz_{H} and zLz_{L}) as initialization for the next forward pass. This allows the model to reason over many iterations and improve its latent features (zLz_{L} and zHz_{H}) until it (hopefully) converges to the correct solution. At most Ns​u​p=16N_{sup}=16 supervision steps are used.

2.5 Adaptive computational time (ACT)

With deep supervision, each mini-batch of data samples must be used for Ns​u​p=16N_{sup}=16 supervision steps before moving to the next mini-batch. This is expensive, and there is a balance to be reached between optimizing a few data examples for many supervision steps versus optimizing many data examples with less supervision steps. To reach a better balance, a halting mechanism is incorporated to determine whether the model should terminate early. It is learned through a Q-learning objective that requires passing the zHz_{H} through an additional head and running an additional forward pass (to determine if halting now rather than later would have been preferable). They call this method Adaptive computational time (ACT). It is only used during training, while the full Ns​u​p=16N_{sup}=16 supervision steps are done at test time to maximize downstream performance. ACT greatly diminishes the time spent per example (on average spending less than 2 steps on the Sudoku-Extreme dataset rather than the full Ns​u​p=16N_{sup}=16 steps), allowing more coverage of the dataset given a fixed number of training iterations.

2.6 Deep supervision and 1-step gradient approximations replaces BPTT

Deep supervision and the 1-step gradient approximation provide a more biologically plausible and less computationally-expansive alternative to Backpropagation Through Time (BPTT) (Werbos, 1974; Rumelhart et al., 1985; LeCun, 1985) for solving the temporal credit assignment (TCA) (Rumelhart et al., 1985; Werbos, 1988; Elman, 1990) problem (Lillicrap & Santoro, 2019). The implication is that HRM can learn what would normally require an extremely large network without having to back-propagate through its entire depth. Given the hyperparameters used by Jang et al. (2023) in all their experiments, HRM effectively reasons over nl​a​y​e​r​s​(n+1)​T​Ns​u​p=4∗(2+1)∗2∗16=384n_{layers}(n+1)TN_{sup}=4*(2+1)*2*16=384 layers of effective depth.

2.7 Summary of HRM

HRM leverages recursion from two networks at different frequencies (high frequency versus low frequency) and deep supervision to learn to improve its answer over multiple supervision steps (with ACT to reduce time spent per data example). This enables the model to imitate extremely large depth without requiring backpropagation through all layers. This approach obtains significantly higher performance on hard question-answer tasks that regular supervised models struggle with. However, this method is quite complicated, relying a bit too heavily on uncertain biological arguments and fixed-point theorems that are not guaranteed to be applicable. In the next section, we discuss those issues and potential targets for improvements in HRM.

3 Target for improvements in Hierarchical Reasoning Models

In this section, we identify key targets for improvements in HRM, which will be addressed by our proposed method, Tiny Recursion Models (TRMs).

3.1 Implicit Function Theorem (IFT) with 1-step gradient approximation

HRM only back-propagates through the last 2 of the 6 recursions. The authors justify this by leveraging the Implicit Function Theorem (IFT) and one-step approximation, which states that when a recurrent function converges to a fixed point, backpropagation can be applied in a single step at that equilibrium point.

There are concerns about applying this theorem to HRM. Most importantly, there is no guarantee that a fixed-point is reached. Deep equilibrium models normally do fixed-point iteration to solve for the fixed pointz∗=f​(z∗)z^{*}=f(z^{*}) (Bai et al., 2019). However, in the case of HRM, it is not iterating to the fixed-point but simply doing forward passes of fLf_{L} and fHf_{H}. To make matters worse, HRM is only doing 4 recursions before stopping to apply the one-step approximation. After its first loop of two fLf_{L} and 1 fHf_{H} evaluations, it only apply a single fLf_{L} evaluation before assuming that a fixed-point is reached for both zLz_{L} and zHz_{H} (zL∗=fL​(zL∗+zH+x)z_{L}^{{*}=f_{L}(z_{L}}{*}+z_{H}+x) and zH∗=fH​(zL∗+zH∗)z_{H}^{{*}=f_{H}(z_{L}}{*}+z_{H}^{*})). Then, the one-step gradient approximation is applied to both latent variables in succession.

The authors justify that a fixed-point is reached by depicting an example with n=7n=7 and T=7T=7 where the forward residuals is reduced over time (Figure 3 in Wang et al. (2025)). Even in this setting, which is different from the much smaller n=2n=2 and T=2T=2 used in every experiment of their paper, we observe the following:

1. 1.

the residual for zHz_{H} is clearly well above 0 at every step

1. 2.

the residual for zLz_{L} only becomes closer to 0 after many cycles, but it remains significantly above 0

1. 3.

zLz_{L} is very far from converged after one fLf_{L} evaluation at TT cycles, which is when the fixed-point is assumed to be reached and the 1-step gradient approximation is used

Thus, while the application of the IFT theorem and 1-step gradient approximation to HRM has some basis since the residuals do generally reduce over time, a fixed point is unlikely to be reached when the theorem is actually applied.

In the next section, we show that we can bypass the need for the IFT theorem and 1-step gradient approximation, thus bypassing the issue entirely.

3.2 Twice the forward passes with Adaptive computational time (ACT)

HRM uses Adaptive computational time (ACT) during training to optimize the time spent of each data sample. Without ACT, Ns​u​p=16N_{sup}=16 supervision steps would be spent on the same data sample, which is highly inefficient. They implement ACT through an additional Q-learning objective, which decides when to halt and move to a new data sample rather than keep iterating on the same data. This allows much more efficient use of time especially since the average number of supervision steps during training is quite low with ACT (less than 2 steps on the Sudoku-Extreme dataset as per their reported numbers).

However, ACT comes at a cost. This cost is not directly shown in the HRM’s paper, but it is shown in their official code. The Q-learning objective relies on a halting loss and a continue loss. The continue loss requires an extra forward pass through HRM (with all 6 function evaluations). This means that while ACT optimizes time more efficiently per sample, it requires 2 forward passes per optimization step. The exact formulation is shown in Algorithm 2.

In the next section, we show that we can bypass the need for two forward passes in ACT.

3.3 Hierarchical interpretation based on complex biological arguments

The HRM’s authors justify the two latent variables and two networks operating at different hierarchies based on biological arguments, which are very far from artificial neural networks. They even try to match HRM to actual brain experiments on mice. While interesting, this sort of explanation makes it incredibly hard to parse out why HRM is designed the way it is. Given the lack of ablation table in their paper, the over-reliance on biological arguments and fixed-point theorems (that are not perfectly applicable), it is hard to determine what parts of HRM is helping what and why. Furthermore, it is not clear why they use two latent features rather than other combinations of features.

In the next section, we show that the recursive process can be greatly simplified and understood in a much simpler manner that does not require any biological argument, fixed-point theorem, hierarchical interpretation, nor using two networks. It also explains why 2 is the optimal number of features (zLz_{L} and zHz_{H}).

def latent_recursion(x, y, z, n=6):

for i in range(n):

z = net(x, y, z)

y = net(y, z)

return y, z

def deep_recursion(x, y, z, n=6, T=3):

with torch.no_grad():

for j in range(T-1):

y, z = latent_recursion(x, y, z, n)

y, z = latent_recursion(x, y, z, n)

return (y.detach(), z.detach()), output_head(y), Q_head(y)

for x_input, y_true in train_dataloader:

y, z = y_init, z_init

for step in range(N_supervision):

x = input_embedding(x_input)

(y, z), y_hat, q_hat = deep_recursion(x, y, z)

loss = softmax_cross_entropy(y_hat, y_true)

loss += binary_cross_entropy(q_hat, (y_hat == y_true))

loss.backward()

opt.step()

opt.zero_grad()

if q_hat > 0:

break

Figure 3: Pseudocode of Tiny Recursion Models (TRMs).

4 Tiny Recursion Models

In this section, we present Tiny Recursion Models (TRMs). Contrary to HRM, TRM requires no complex mathematical theorem, hierarchy, nor biological arguments. It generalizes better while requiring only a single tiny network (instead of two medium-size networks) and a single forward pass for the ACT (instead of 2 passes). Our approach is described in Algorithm 3 and illustrated in Figure 1. We also provide an ablation in Table 1 on the Sudoku-Extreme dataset (a dataset of difficult Sudokus with only 1K training examples, but 423K test examples). Below, we explain the key components of TRMs.

Table 1: Ablation of TRM on Sudoku-Extreme comparing % Test accuracy, effective depth per supervision step (T​(n+1)​nl​a​y​e​r​s)(T(n+1)n_{layers}), number of Forward Passes (NFP) per optimization step, and number of parameters

4.1 No fixed-point theorem required

HRM assumes that the recursions converge to a fixed-point for both zLz_{L} and zHz_{H} in order to leverage the 1-step gradient approximation (Bai et al., 2019). This allows the authors to justify only back-propagating through the last two function evaluations (1 fLf_{L} and 1 fHf_{H}). To bypass this theoretical requirement, we define a full recursion process as containing nn evaluations of fLf_{L} and 1 evaluation of fHf_{H}:

	zL\displaystyle z_{L}	←fL​(zL+zH+x)\displaystyle\leftarrow f_{L}\left(z_{L}+z_{H}+x\right)
		…\displaystyle...
	zL\displaystyle z_{L}	←fL​(zL+zH+x)\displaystyle\leftarrow f_{L}\left(z_{L}+z_{H}+x\right)
	zH\displaystyle z_{H}	←fH​(zL+zH).\displaystyle\leftarrow f_{H}\left(z_{L}+z_{H}\right).

Then, we simply back-propagate through the full recursion process.

Through deep supervision, the models learns to take any (zL,zH)(z_{L},z_{H}) and improve it through a full recursion process, hopefully making zHz_{H} closer to the solution. This means that by the design of the deep supervision goal, running a few full recursion processes (even without gradients) is expected to bring us closer to the solution. We propose to run T−1T-1 recursion processes without gradient to improve (zL,zH)(z_{L},z_{H}) before running one recursion process with backpropagation.

Thus, instead of using the 1-step gradient approximation, we apply a full recursion process containing nn evaluations of fLf_{L} and 1 evaluation of fHf_{H}. This removes entirely the need to assume that a fixed-point is reached and the use of the IFT theorem with 1-step gradient approximation. Yet, we can still leverage multiple backpropagation-free recursion processes to improve (zL,zH)(z_{L},z_{H}). With this approach, we obtain a massive boost in generalization on Sudoku-Extreme (improving TRM from 56.5% to 87.4%; see Table 1).

4.2 Simpler reinterpretation of zHz_{H} and zLz_{L}

HRM is interpreted as doing hierarchical reasoning over two latent features of different hierarchies due to arguments from biology. However, one might wonder why use two latent features instead of 1, 3, or more? And do we really need to justify these so-called ”hierarchical” features based on biology to make sense of them? We propose a simple non-biological explanation, which is more natural, and directly answers the question of why there are 2 features.

The fact of the matter is: zHz_{H} is simply the current (embedded) solution. The embedding is reversed by applying the output head and rounding to the nearest token using the argmax operation. On the other hand, zLz_{L} is a latent feature that does not directly correspond to a solution, but it can be transformed into a solution by applying zH←fH​(x,zL,zH)z_{H}\leftarrow f_{H}(x,z_{L},z_{H}). We show an example on Sudoku-Extreme in Figure 6 to highlight the fact that zHz_{H} does correspond to the solution, but zLz_{L} does not.

Once this is understood, hierarchy is not needed; there is simply an input xx, a proposed solution yy (previously called zHz_{H}), and a latent reasoning feature zz (previously called zLz_{L}). Given the input question xx, current solution yy, and current latent reasoning zz, the model recursively improves its latent zz. Then, given the current latent zz and the previous solution yy, the model proposes a new solution yy (or stay at the current solution if its already good).

Although this has no direct influence on the algorithm, this re-interpretation is much simpler and natural. It answers the question about why two features: remembering in context the question xx, previous reasoning zz, and previous answer yy helps the model iterate on the next reasoning zz and then the next answer yy. If we were not passing the previous reasoning zz, the model would forget how it got to the previous solution yy (since zz acts similarly as a chain-of-thought). If we were not passing the previous solution yy, then the model would forget what solution it had and would be forced to store the solution yy within zz instead of using it for latent reasoning. Thus, we need both yy and zz separately, and there is no apparent reason why one would need to split zz into multiple features.

While this is intuitive, we wanted to verify whether using more or less features could be helpful. Results are shown in Table 2.

More features (>2>2): We tested splitting zz into different features by treating each of the nn recursions as producing a different ziz_{i} for i=1,…,ni=1,...,n. Then, each ziz_{i} is carried across supervision steps. The approach is described in Algorithm 5. In doing so, we found performance to drop. This is expected because, as discussed, there is no apparent need for splitting zz into multiple parts. It does not have to be hierarchical.

Single feature: Similarly, we tested the idea of taking a single feature by only carrying zHz_{H} across supervision steps. The approach is described in Algorithm 4. In doing so, we found performance to drop. This is expected because, as discussed, it forces the model to store the solution yy within zz.

Thus, we explored using more or less latent variables on Sudoku-Extreme, but found that having only yy and zz lead to better test accuracy in addition to being the simplest more natural approach.

Table 2: TRM on Sudoku-Extreme comparing % Test accuracy when using more or less latent features

4.3 Single network

HRM uses two networks, one applied frequently as a low-level module fHf_{H} and one applied rarely as an high-level module (fHf_{H}). This requires twice the number of parameters compared to regular supervised learning with a single network.

As mentioned previously, while fLf_{L} iterates on the latent reasoning feature zz (zLz_{L} in HRM), the goal of fHf_{H} is to update the solution yy (zHz_{H} in HRM) given the latent reasoning and current solution. Importantly, since z←fL​(x+y+z)z\leftarrow f_{L}(x+y+z) contains xx but y←fH​(y+z)y\leftarrow f_{H}(y+z) does not contains xx, the task to achieve (iterating on zz versus using zz to update yy) is directly specified by the inclusion or lack of xx in the inputs. Thus, we considered the possibility that both networks could be replaced by a single network doing both tasks. In doing so, we obtain better generalization on Sudoku-Extreme (improving TRM from 82.4% to 87.4%; see Table 1) while reducing the number of parameters by half. It turns out that a single network is enough.

4.4 Less is more

We attempted to increase capacity by increasing the number of layers in order to scale the model. Surprisingly, we found that adding layers decreased generalization due to overfitting. In doing the opposite, decreasing the number of layers while scaling the number of recursions (nn) proportionally (to keep the amount of compute and emulated depth approximately the same), we found that using 2 layers (instead of 4 layers) maximized generalization. In doing so, we obtain better generalization on Sudoku-Extreme (improving TRM from 79.5% to 87.4%; see Table 1) while reducing the number of parameters by half (again).

It is quite surprising that smaller networks are better, but 2 layers seems to be the optimal choice. Bai & Melas-Kyriazi (2024) also observed optimal performance for 2-layers in the context of deep equilibrium diffusion models; however, they had similar performance to the bigger networks, while we instead observe better performance with 2 layers. This may appear unusual, as with modern neural networks, generalization tends to directly correlate with model sizes. However, when data is too scarce and model size is large, there can be an overfitting penalty (Kaplan et al., 2020). This is likely an indication that there is too little data. Thus, using tiny networks with deep recursion and deep supervision appears to allow us to bypass a lot of the overfitting.

4.5 attention-free architecture for tasks with small fixed context length

Self-attention is particularly good for long-context lengths when L≫DL\gg D since it only requires a matrix of [D,3​D][D,3D] parameters, even though it can account for the whole sequence. However, when focusing on tasks where L≤DL\leq D, a linear layer is cheap, requiring only a matrix of [L,L][L,L] parameters. Taking inspiration from the MLP-Mixer (Tolstikhin et al., 2021), we can replace the self-attention layer with a multilayer perceptron (MLP) applied on the sequence length. Using an MLP instead of self-attention, we obtain better generalization on Sudoku-Extreme (improving from 74.7% to 87.4%; see Table 1). This worked well on Sudoku 9x9 grids, given the small and fixed context length; however, we found this architecture to be suboptimal for tasks with large context length, such as Maze-Hard and ARC-AGI (both using 30x30 grids). We show results with and without self-attention for all experiments.

4.6 No additional forward pass needed with ACT

As previously mentioned, the implementation of ACT in HRM through Q-learning requires two forward passes, which slows down training. We propose a simple solution, which is to get rid of the continue loss (from the Q-learning) and only learn a halting probability through a Binary-Cross-Entropy loss of having reached the correct solution. By removing the continue loss, we remove the need for the expensive second forward pass, while still being able to determine when to halt with relatively good accuracy. We found no significant difference in generalization from this change (going from 86.1% to 87.4%; see Table 1).

4.7 Exponential Moving Average (EMA)

On small data (such as Sudoku-Extreme and Maze-Hard), HRM tends to overfit quickly and then diverge. To reduce this problem and improves stability, we integrate Exponential Moving Average (EMA) of the weights, a common technique in GANs and diffusion models to improve stability (Brock et al., 2018; Song & Ermon, 2020). We find that it prevents sharp collapse and leads to higher generalization (going from 79.9% to 87.4%; see Table 1).

4.8 Optimal the number of recursions

We experimented with different number of recursions by varying TT and nn and found that T=3,n=3T=3,n=3 (equivalent to 48 recursions) in HRM and T=3,n=6T=3,n=6 in TRM (equivalent to 42 recursions) to lead to optimal generalization on Sudoku-Extreme. More recursions could be helpful for harder problems (we have not tested it, given our limited resources); however, increasing either TT or nn incurs massive slowdowns. We show results at different nn and TT for HRM and TRM in Table 3. Note that TRM requires backpropagation through a full recursion process, thus increasing nn too much leads to Out Of Memory (OOM) errors. However, this memory cost is well worth its price in gold.

Table 3: % Test accuracy on Sudoku-Extreme dataset. HRM versus TRM matched at a similar effective depth per supervision step (T​(n+1)​nl​a​y​e​r​s)(T(n+1)n_{layers})

In the following section, we show our main results on multiple datasets comparing HRM, TRM, and LLMs.

5 Results

Following Wang et al. (2025), we test our approach on the following datasets: Sudoku-Extreme (Wang et al., 2025), Maze-Hard (Wang et al., 2025), ARC-AGI-1 (Chollet, 2019) and, ARC-AGI-2 (Chollet et al., 2025). Results are presented in Tables 4 and 5. Hyperparameters are detailed in Section Hyper-parameters and setup. Datasets are discussed below.

Sudoku-Extreme consists of extremely difficult Sudoku puzzles (Dillion, 2025; Palm et al., 2018; Park, 2018) (9x9 grid), for which only 1K training samples are used to test small-sample learning. Testing is done on 423K samples. Maze-Hard consists of 30x30 mazes generated by the procedure by Lehnert et al. (2024) whose shortest path is of length above 110; both the training set and test set include 1000 mazes.

ARC-AGI-1 and ARC-AGI-2 are geometric puzzles involving monetary prizes. Each puzzle is designed to be easy for a human, yet hard for current AI models. Each puzzle task consists of 2-3 input–output demonstration pairs and 1-2 test inputs to be solved. The final score is computed as the accuracy over all test inputs from two attempts to produce the correct output grid. The maximum grid size is 30x30. ARC-AGI-1 contains 800 tasks, while ARC-AGI-2 contains 1120 tasks. We also augment our data with the 160 tasks from the closely related ConceptARC dataset (Moskvichev et al., 2023). We provide results on the public evaluation set for both ARC-AGI-1 and ARC-AGI-2.

While these datasets are small, heavy data-augmentation is used in order to improve generalization. Sudoku-Extreme uses 1000 shuffling (done without breaking the Sudoku rules) augmentations per data example. Maze-Hard uses 8 dihedral transformations per data example. ARC-AGI uses 1000 data augmentations (color permutation, dihedral-group, and translations transformations) per data example. The dihedral-group transformations consist of random 90-degree rotations, horizontal/vertical flips, and reflections.

From the results, we see that TRM without self-attention obtains the best generalization on Sudoku-Extreme (87.4% test accuracy). Meanwhile, TRM with self-attention generalizes better on the other tasks (probably due to inductive biases and the overcapacity of the MLP on large 30x30 grids). TRM with self-attention obtains 85.3% accuracy on Maze-Hard, 44.6% accuracy on ARC-AGI-1, and 7.8% accuracy on ARC-AGI-2 with 7M parameters. This is significantly higher than the 74.5%, 40.3%, and 5.0% obtained by HRM using 4 times the number of parameters (27M).

Table 4: % Test accuracy on Puzzle Benchmarks (Sudoku-Extreme and Maze-Hard) Table 5: % Test accuracy on ARC-AGI Benchmarks (2 tries)

6 Conclusion

We propose Tiny Recursion Models (TRM), a simple recursive reasoning approach that achieves strong generalization on hard tasks using a single tiny network recursing on its latent reasoning feature and progressively improving its final answer. Contrary to the Hierarchical Reasoning Model (HRM), TRM requires no fixed-point theorem, no complex biological justifications, and no hierarchy. It significantly reduces the number of parameters by halving the number of layers and replacing the two networks with a single tiny network. It also simplifies the halting process, removing the need for the extra forward pass. Overall, TRM is much simpler than HRM, while achieving better generalization.

While our approach led to better generalization on 4 benchmarks, every choice made is not guaranteed to be optimal on every dataset. For example, we found that replacing the self-attention with an MLP worked extremely well on Sudoku-Extreme (improving test accuracy by 10%), but poorly on other datasets. Different problem settings may require different architectures or number of parameters. Scaling laws are needed to parametrize these networks optimally. Although we simplified and improved on deep recursion, the question of why recursion helps so much compared to using a larger and deeper network remains to be explained; we suspect it has to do with overfitting, but we have no theory to back this explaination. Not all our ideas made the cut; we briefly discuss some of the failed ideas that we tried but did not work in Section Ideas that failed. Currently, recursive reasoning models such as HRM and TRM are supervised learning methods rather than generative models. This means that given an input question, they can only provide a single deterministic answer. In many settings, multiple answers exist for a question. Thus, it would be interesting to extend TRM to generative tasks.

Acknowledgements

Thank you Emy Gervais for your invaluable support and extra push. This research was enabled in part by computing resources, software, and technical assistance provided by Mila and the Digital Research Alliance of Canada.

References

• ARC Prize Foundation (2025a) ARC Prize Foundation. The Hidden Drivers of HRM’s Performance on ARC-AGI. https://arcprize.org/blog/hrm-analysis, 2025a. [Online; accessed 2025-09-15].
• ARC Prize Foundation (2025b) ARC Prize Foundation. ARC-AGI Leaderboard. https://arcprize.org/leaderboard, 2025b. [Online; accessed 2025-09-24].
• Bai et al. (2019) Bai, S., Kolter, J. Z., and Koltun, V. Deep equilibrium models. Advances in neural information processing systems, 32, 2019.
• Bai & Melas-Kyriazi (2024) Bai, X. and Melas-Kyriazi, L. Fixed point diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9430–9440, 2024.
• Brock et al. (2018) Brock, A., Donahue, J., and Simonyan, K. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv:1809.11096, 2018.
• Chollet (2019) Chollet, F. On the measure of intelligence. arXiv preprint arXiv:1911.01547, 2019.
• Chollet et al. (2025) Chollet, F., Knoop, M., Kamradt, G., Landers, B., and Pinkard, H. Arc-agi-2: A new challenge for frontier ai reasoning systems. arXiv preprint arXiv:2505.11831, 2025.
• Chowdhery et al. (2023) Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1–113, 2023.
• Dillion (2025) Dillion, T. Tdoku: A fast sudoku solver and generator. https://t-dillon.github.io/tdoku/, 2025.
• Elman (1990) Elman, J. L. Finding structure in time. Cognitive science, 14(2):179–211, 1990.
• Fedus et al. (2022) Fedus, W., Zoph, B., and Shazeer, N. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1–39, 2022.
• Geng & Kolter (2023) Geng, Z. and Kolter, J. Z. Torchdeq: A library for deep equilibrium models. arXiv preprint arXiv:2310.18605, 2023.
• Hendrycks & Gimpel (2016) Hendrycks, D. and Gimpel, K. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415, 2016.
• Jang et al. (2023) Jang, Y., Kim, D., and Ahn, S. Hierarchical graph generation with k2-trees. In ICML 2023 Workshop on Structured Probabilistic Inference Generative Modeling, 2023.
• Kaplan et al. (2020) Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.
• Kingma & Ba (2014) Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
• Krantz & Parks (2002) Krantz, S. G. and Parks, H. R. The implicit function theorem: history, theory, and applications. Springer Science & Business Media, 2002.
• LeCun (1985) LeCun, Y. Une procedure d’apprentissage ponr reseau a seuil asymetrique. Proceedings of cognitiva 85, pp. 599–604, 1985.
• Lehnert et al. (2024) Lehnert, L., Sukhbaatar, S., Su, D., Zheng, Q., Mcvay, P., Rabbat, M., and Tian, Y. Beyond a*: Better planning with transformers via search dynamics bootstrapping. arXiv preprint arXiv:2402.14083, 2024.
• Lillicrap & Santoro (2019) Lillicrap, T. P. and Santoro, A. Backpropagation through time and the brain. Current opinion in neurobiology, 55:82–89, 2019.
• Loshchilov & Hutter (2017) Loshchilov, I. and Hutter, F. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
• Moskvichev et al. (2023) Moskvichev, A., Odouard, V. V., and Mitchell, M. The conceptarc benchmark: Evaluating understanding and generalization in the arc domain. arXiv preprint arXiv:2305.07141, 2023.
• Palm et al. (2018) Palm, R., Paquet, U., and Winther, O. Recurrent relational networks. Advances in neural information processing systems, 31, 2018.
• Park (2018) Park, K. Can convolutional neural networks crack sudoku puzzles? https://github.com/Kyubyong/sudoku, 2018.
• Prieto et al. (2025) Prieto, L., Barsbey, M., Mediano, P. A., and Birdal, T. Grokking at the edge of numerical stability. arXiv preprint arXiv:2501.04697, 2025.
• Rumelhart et al. (1985) Rumelhart, D. E., Hinton, G. E., and Williams, R. J. Learning internal representations by error propagation. Technical report, 1985.
• Shazeer (2020) Shazeer, N. Glu variants improve transformer. arXiv preprint arXiv:2002.05202, 2020.
• Shazeer et al. (2017) Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., and Dean, J. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538, 2017.
• Snell et al. (2024) Snell, C., Lee, J., Xu, K., and Kumar, A. Scaling llm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314, 2024.
• Song & Ermon (2020) Song, Y. and Ermon, S. Improved techniques for training score-based generative models. Advances in neural information processing systems, 33:12438–12448, 2020.
• Su et al. (2024) Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., and Liu, Y. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024.
• Tolstikhin et al. (2021) Tolstikhin, I. O., Houlsby, N., Kolesnikov, A., Beyer, L., Zhai, X., Unterthiner, T., Yung, J., Steiner, A., Keysers, D., Uszkoreit, J., et al. Mlp-mixer: An all-mlp architecture for vision. Advances in neural information processing systems, 34:24261–24272, 2021.
• Vaswani et al. (2017) Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. Attention is all you need. Advances in neural information processing systems, 30, 2017.
• Wang et al. (2025) Wang, G., Li, J., Sun, Y., Chen, X., Liu, C., Wu, Y., Lu, M., Song, S., and Yadkori, Y. A. Hierarchical reasoning model. arXiv preprint arXiv:2506.21734, 2025.
• Wei et al. (2022) Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022.
• Werbos (1974) Werbos, P. Beyond regression: New tools for prediction and analysis in the behavioral sciences. PhD thesis, Committee on Applied Mathematics, Harvard University, Cambridge, MA, 1974.
• Werbos (1988) Werbos, P. J. Generalization of backpropagation with application to a recurrent gas market model. Neural networks, 1(4):339–35