Published on November 3, 2025 4:10 PM GMT

Introduction

• Basis: The rationalist project was built on the idea of overcoming bias, using the last half-century of psych findings to eliminate errors and thereby become stronger.
• Problem: The psych literature is wrong (mostly, probably, plausibly; we don’t even know).
• Problem: Even when it’s right, it might not apply to an individual: maybe people in general are overconfident, but that doesn’t mean you specifically are on the specific topic in question.
• Problem: Some biases are subtly loadbearing, or cancel out other biases.
• But: We do keep getting important things right, especially purely-epistemically^[1]. We called crypto, called covid, and called AI. (If I’d made investment decisions over the last decade based solely on What LW Is On About This Month, I’d be retired by now.)
• Also: We’re one of the few spaces which have even started to process the massive holes in the psych literature, or who collectively grok statistics well enough to understand where they went wrong.
• And: The study of what minds can do seems easier than the study of what minds will do. Test subjects, incentivized appropriately, will consistently try their best; we can then study what conditions improve their best.
• So: I don’t think we get to give up.

My proposed approach

Put people in synthetic situations where the problems you’re warning people against are anomalously but legitimately salient.

Notes:

• Errors are best addressed en masse. If you deliver a 30 minute class about avoiding the sunk cost fallacy, they will - if you’re lucky - learn to avoid the sunk cost fallacy in the context of a 30 minute class about avoiding the sunk cost fallacy. If they don’t know in advance which errors will hurt them or how, and it’s not immediately obvious from the shape of the challenge, that should give Transfer Learning a fighting chance.
• You should account for the possibility that people are making the opposite mistake. It makes sense to tilt the experience so it’s about the more common flaw, but if someone is chronically underconfident or changes tack too readily, they shouldn’t be rewarded for that. (Maintaining some level of agnosticism here also reduces dependency on the psych literature.)
• “Figure this out” is good. “Figure this out and then use it” is better. (c.f. natural/compound weightlifts vs weightlifting on machines)

Isn’t this just videogames? Or games in general?

No. It should be. But it isn’t.

Games:

• Are usually optimized for fun, not challenge.
  • Are, when optimized for challenge, usually optimized for challenge-for-the-sake-of-challenge, not challenge-for-the-sake-of-getting-stronger-outside-the-game.
• Are usually not primarily about strategy.
  • Are, when strategic, usually too easy to teach the average asp-rat anything.
    • Are, when strategic and difficult, usually dependent on massive time commitments.
      • (. . . like only getting to lift heavy weights with high numbers of reps; like having your only choices being light jogs or gruelling marathons, and never getting to sprint.)
• Are usually not about Inference.
  • Are, when about Inference, usually center maximally-fuzzy vibes and/or maximally-sterile logic; few intermediate contexts where e.g. Bayes is appropriate.
• Will actively lie to you about probability in particular. (XCOM:EU is the one modern-ish mainstream-ish hard strategic uncertainty-involving title I’m aware of where the probabilities shown onscreen are the actual probabilities; everything else including the sequel systematically lies to the player to accommodate their limitations and mitigate their frustrations.^[2])
• Require genre familiarity (to an extent you’re probably ignoring if you have genre familiarity).

Isn’t this just life?

Nope.

Life:

• Doesn’t give you multiple tries^[3].
• Doesn’t show all its rules at the end so you know where you went wrong^[3].
• Has long feedback loops.
• Presents important decisions infrequently and suddenly.
• Has high stakes which complicate learning from big mistakes: it’s hard to admit you screwed up something important, and trauma can make you over-correct^[4].
• Has all the easy and fundamental inferential problems already solved; you need to spend at least two decades studying things other people figured out before you have a chance to discover something novel and important and timeless, which will be your first chance to use or develop your discover-something-novel-and-important-and-timeless skills.
• Is notoriously unfair and arbitrary^[3].

. . . Is this a “please make and/or play more D&D.Sci scenarios you guys” post?

I wish!

Despite its merits, my genre:

• Hard-requires and focuses-on data science skills; rationality skills are secondary. (People who don’t know their way around a command line or a spreadsheet are at best stuck spectating.)
• Doesn’t emotionally involve players, except in a “sure hope I’m right!”, “argh I don’t understand!”, or a “wonder what the gimmick is this time?” sense; no chance to practice managing or channeling Feels during problem-solving.
• Requires a (honestly, probably excessively) significant investment of time and effort.
• Hard-requires and focuses-on data science skills; rationality skills are secondary. (I know I said this before but it’s a significant enough limit that imo it’s worth repeating.)

So what’s the plan?

Well, in the immediate term, my plan . . .

. . . is to make this post, and then hope someone else follows up on it.

I do have some ideas for rationalish/inferencey games^[5], but I’m starting a demanding new dayjob next week and I expect that to be my priority for the foreseeable; realistically you’re not going to get anything solid from me until the middle of next year at the earliest. So if anyone wants a headstart on me, they’re warmly encouraged to begin building now^[6].

(Also, if anyone knows about already-existing rationality games I might have missed, I look forward to seeing them in the comments.)

1. Conversely, most of our failures seem to happen at the “And What Should We Do About That?” step.

2. I reserve a special place in hell for Fire Emblem’s numerical clownery.

3. NB: this claim is contested.

4. These aren’t mutually exclusive; people can be in denial about what they actually got wrong, and then over-correct for something else.

5. This is actually an understatement on par with “Mt. Everest is somewhat taller than the average house”.

6. . . . and to DM me for support playtesting: I’m told I give helpful feedback!