Published on November 21, 2025 1:06 AM GMT

Season Recap

Human thinking has multiple layers of abstraction. Just as walking can be analyzed at the levels of “walk from A to B”, “angles at which which knee, hip, and ankle joints bend”, “individual muscle contractions”, and “signaling to muscle fibres”, so does thinking decompose from high-level questions like “what should I have for lunch?” into smaller mental operations like thinking of options (“tuna”, “pasta”, “takeout”) and considerations (“healthy”, “tasty”, “economical”).

As with walking, the lower levels aren’t something we consciously think about, or can think about. When’s the last time you thought about the angle you’re bending your knees at when walking?

These realities mean that human thinking both has many degrees of freedom to vary, at many levels, and that we fail to notice just how varied human thinking could be. The lack of conscious awareness within ourselves of all the parts of our thinking compounds with the invisibility of other people’s thinking. In the third post of this series, I argued we likely underestimate our differences from others when those differences are private.

In the second post, I asserted that major differences between people arise not just from what mental motions they can theoretically execute, but from the patterns and habits of thoughts they’ve learned.

The point of this post is to explore why people end up learning some different ways of thinking.

Genetics / Phenotype isn’t a complete explanation

Plausibly, we might reason, people think differently because we actually have different brains. Just genetically, people are different. Maybe this could have been true; I don’t think it is. One intuition pump here is that people vary in size, muscle mass, body-part length ratios, etc., and yet people mostly walk the same. Trained people can do gait analysis, but apart from the occasional person with swagger, I don’t think we notice much difference in how people walk. There’s just like an approximately optimal way, and we all do that.

So if thinking followed the same pattern, while some people would be better at it, the way some people walk faster and more capably, we’d be doing the same thing.

This isn’t a knockdown argument: brain phenotypes might differ more than body phenotypes relevant to walking. Still, I don’t think it’s all that’s going on.

It’s learned, and it’s all about reward

You might think that walking is something that evolution could figure out and completely encode in the genome for the brain. But it doesn’t. Even animals that are able to walk very soon after birth do so with some stumbling around and practice, and improve in the following days and weeks. Certainly, for humans, each of us individually learn walking via trial and error, iteration hones the neural pathways for motor control until it is learned, automatic, and subconscious.

It’s essential here that it is learning from feedback. The infant has a clear target of crossing the floor bipedally and an anti-target of falling on the floor. They try repeatedly, and they learn.

I don’t think we have any reason to think that thinking is any different. It’s neither the case that we are born fully able to think (clearly not) nor the case that, as the brain matures, upon hitting certain milestones, new mental motions are suddenly unlocked. Rather, like walking, there’s a gradual process of trying to accomplish certain goals and seeing which mental motions work. Ultimately, they end up learned: and very much automatic and subconscious except at the top-level (the way we still consciously intend to walk from here to there but don’t think about foot placement).

Forgive me for repeating the following; it’s important. Thinking is different in a couple of ways, though. (1) The reward signal is much more complicated than that for walking. Vastly more. There are all kinds of things our thinking can get us, such as approval from others, resources we like, experiences we like, inherently satisfying experiences like sating curiosity, and so on. (2) The range of mental motions is much wider than the range of leg motions.

The key thing is that little human minds come into the world not knowing which mental motions will work, but through trial and error, they find ones that work for them. Walking is maybe too simple. We could analogize to various sports. Playing tennis. One starts with crude motions, but over time, one masters a technique^[1]. That technique may or may not be globally optimal. It may or may not generalize well from where it’s learned. People do learn bad habits in sports. The same in thought.

We didn’t all get the same reward signals, even if we wanted the same things

Human children^[2] have very similar needs to human adults. We want food, safety, entertainment, social acceptance, maybe even love and attention. Some of what works to get these rewards is dependent on environment, e.g., learning which plants and animals are safe to eat. A lot of what works to meet wants in need, especially in modern childhood, routes through the approval and favor of parents, teachers, and peers. 

Environmental rewards and social rewards differ. I’ll start from the obvious and move towards the less so.

Environmental Differences

My mother tells a story of her younger brother’s friend who would come over to play. One time, my grandmother took him and my uncle down to the shops and left them in the car while she popped in to buy something. The friend burst into tears, distraught. He’d immigrated with his family from the Soviet Union, and he expected going to the shops to take hours.

Take children of any genes, and I think you can cause them to be more anxious, more food-focused, more scarcity-minded, if you place them in environments where those kinds of temperaments are beneficial.

Learning to play to your relative strengths

My thinking here is deeply downstream of Kevin Simler’s excellent Personality: The Body in Society.

Though much of our final loadout is determined through development, we are born with certain dispositions: a natural talent for running, math, oration, or a naturally beautiful face. Notwithstanding, personality can’t be packaged deterministically with other traits. Whether it’s a good strategy (and personality is strategy^[3]) to be aggressive and domineering is dependent not on your muscles, but on your muscles relative to everyone else’s^[4]. This is something one learns. Pick a few fights, profit and lose horribly, and tendencies will get reinforced.

This goes just as much for the kinds of cognition you have. You’re funnier than everyone else? You’ll learn to make jokes. Poetically gifted? You’ll probably double down on that skill. And so on.

What do your parents, peers, and cultures reward?

Even if you’re not athletically gifted, if your parents and peers and all those around you venerate athletic accomplishment, there’s a good chance you’ll invest in it. The drive for social approval is enough to get people to do a lot of things. And not just to do them, but to absorb that they are good.

And what those around us reward is more than what they explicitly value. The angry, explosive parent will train unobtrusiveness. The parent who only pays attention to wrongdoing might be training wrongdoing.

All of the above applies to mental motions

There are many types of thinking. At a macro-level, there’s solving puzzles of physical reality (how do I fix this broken thing? how do I build an X that Y’s?), there’s social skills (what is that person thinking? what would most please them?), and there’s these parameters like risk-taking and anxiety levels. 

Throughout life, especially in childhood, these are getting explored and trained. Undoubtedly, there are feedback loops: the child who feels elation at getting an A+ on math (all the more so if parents/peers/culture reinforce that it is goood) will invest effort into repeating that win. The child who didn’t might develop an aversion and instead focus on where their relative strength lies.

To quote Eugene Wei’s excellent Status as a Service (Staas):

Let’s begin with two principles:

• People are status-seeking monkeys*
• People seek out the most efficient path to maximizing social capital

I think the argument is broader than just social capital. In kind of a greedy, local(, myopic?) way, people are efficiently maximizing reward. For different people, the optimal path is different. And this includes patterns of thought.

• Do you focus on “things” vs “people”?^[5]
  • Does one further one’s position more by focusing on “things” or people?
• Does the physical world matter much or does only social politics feel real?
• Are words about reality or are words just rallying cries for your team?
• Do you put numbers on things?
• How much emphasis do you place on wordless felt gut feelings?
• Are you inclined to accept external (especially authoritative/expert beliefs or determined to form your own?)
• Do you track variations in feelings of certainty with regard to the things you believe?
• Do you have a conflict theory or mistake theory frame? Do you assume good faith or arguments war by other means?
• Is it worth trying to come up with System 2 explicit models, or just stick with System 1 instinct?
• Does it feel shameful or admirable to admit you are wrong?
• Does being weird or different from others feel scary or safe?
• Do distant times and places feel real or more like fiction?

I’d guess the full list of relevant traits here is a lot longer. And again, this list is at a high level of abstraction of mental traits because it’s hard to make an ontology of lower mental operations, though I expect those to be varying significantly between people, too.

We can imagine putting each of these traits on a numerical scale. Let’s say -1 to 1. And then position everyone on them. Make a vector in a high-dimensional space. Certainly, there will be correlations. But also a lot of variation across humanity.

I think this variation is there, and I think it’s woefully underappreciated because when someone expresses an opinion on some issue, especially an issue where there are 2-3 mainstream options, you’re not seeing all the weird and wacky ways their mind reached it.

You might imagine they’re running a similar process to you with the truth value of a couple of premises flipped. You make your sandwiches with peanut butter and jelly, and you imagine they’re getting a different result because they’re using Nutella. Actually, they were making tikka masala. You just couldn’t see, because mental behaviors are more private and weirder than shower behaviors.

And this has implications. You might think you’ll get them to agree with you if you can convince them that PB&J is correct and Nutella is bad for you. But they’re not even working with bread! The whole question of condiments is irrelevant. Only due to a real failure of intellectual empathy and poor translation do we think we’re talking about the same things when we use the same words. You think the discussion is sandwiches, they think it’s curry, and everyone is confused by how weird what the other is saying.

I’ll elaborate more on this tomorrow.

Credit to Andrew Critch for this post and series. A chance conversation with him gave me the kernel of  “people get RL’d into different kinds of thought” that I’ve been fleshing out.

1. We might say “mastering some skill” is equivalent to “establish really strong weights in ones brain for effective motions”. (Weights in the ML/AI sense of model parameters.)

2. Not all learning happens in childhood but enough of it that I’m going to frame the learnins here as what happens in youth.

3. Thank you, Kevin Simler.

4. Or possible you learn that muscles aren’t what matters because your society conducts all conflict via chess, and you’re not very good at it.

5. ^{^}

   I know, this is the stereotypical gender split. Perhaps so, I’d bet it’s also attenuated or reinforced by learning.