In the vast swamp of ignorance that has been the debate about AI in the workplace, I’ve slowly been coming to the conclusion…

*There is a large number of people with full time knowledge work jobs that don’t understand the role of intuition. *

We don’t talk much about intuition. It’s the “touchy-feely” stuff. It’s not rational. And yet, intuition is important.

Intuition is the difference between the good waiter and bad waiter. The good waiter knows the customers who are in a hurry, and those who have a lot of time to kill. He can’t tell you why. But experience has taught his intuition: if you provide this for that customer, you’re more likely to get tips.

In Educating Intuition by Robin M Hogarth1Yeah, that’s an affiliate link. Don’t like it. Don’t click it. But if you like the books, would be nice to support this piece of writing by buying from it. My book fund will greatly appreciate it., he describes the lack of study on intuition:2Huge caveat the book was published in 2001, and a lot has changed in that time (Miley Cyrus was 9 and we didn’t yet know Britney Spears was about to have a lousy life). While the recently popular work of Amos Tversky & Daniel Kahneman has investigated how we make unconscious decisions (System 1 thinking), it focuses on the flaws in our reasoning, and how we fool ourselves. Other investigations into intuition focus on mastery and those who spend an ungodly amount of hours carrying out deliberate practice (10,000 hour rule). To my knowledge there is a gap in the middle ground: intuition that comes from everyday experience. Social intelligence in the workplace (and from school) is an example. There have been murmurs about the generational gaps in workplace social intelligence between Gen-Z and Millennials, and I’m sure there are some studies being carried out looking at Covid infants and Covid Graduates. But social intelligence is obvious. What is less obvious: the intuition you get from going to the office, from failure, from watching your peers work etc.

In scientific psychology, intuition has not attracted much attention, and there is hardly a community of scholars dedicated to studying intuition per se. For example, using the American Psychological Association’s “PsycInfo” service to search articles published in scientific journals between 1887 and July 1999 for the keyword “intuition” yields only 2,941 entries. To put that number in perspective, the same search returns 116,108 entries for “attitude,” 72,067 for “similarity,” 22,023 for “instinct,” 10,336 for “causality,” 9,087 for “insight,” 7,614 for “induction,” and 5,253 for “illusion.”

In my day to day job in data, intuition comes with experience. As I explore and read large tables I can start to understand how much to trust the data. Whether it’s good or bad quality. Sometimes it’s obvious — the data dictionary is badly defined, or not been updated in a while. Other times less so: why are they capturing this information when the domain is X? This wasn’t an intentional data capture — just a raw dump from whatever listening method they were using.

Back when my job was more copywriting and less data, I started to develop a gut feel for what was going to land, and what was going to bomb. I remember a data story, which on paper had everything that should have worked, but it just didn’t feel right. I couldn’t put my finger on why. My intuition just said the story would get almost no page views in the long term. My intuition also said—pleasing senior stakeholders was very important at that moment in time. I published.

The story got 1/10th of the views we usually got. It would get single digit views per month while our other stories got 10k+.

The senior stakeholders however were very happy, and didn’t think we were just a bunch of lazy marketers.

What I just described might seem like common sense in the workplace. However, I’m willing to bet that in a pool of 10,000 writers there will be a double digit number of writers who would say you hold off and don’t publish. I also would wager they’d have got fired.

Intuition.

The other day, it was nearing end of day. My brain was fried and I was tired. I wanted to do what felt was an intuitively simple task in SAS3Used to be called “Statistical Analytics Software”. It is still used by large legacy institutions. Like…banks. And I work in one. https://www.sas.com/en_gb/software/base-sas.html either using its native programming language, or SQL: knock out the empty columns from a table. This is easy as p*** for empty rows4WHERE ... IS NULLor equivalent., but not so much for columns. Tired, and really not wanting to wade through SAS documentation, I entered my problem into CoPilot / ChatGPT 5Enterprise approved version, because, again. Big bank. Consumer data protection. Ya get the gist., and I got some seriously convoluted three step solution. I was tired. It was near the end of the week, so I decided to give it a go.

Ran into errors and problems. Problem solved them. Tried again.

Finally had something running. Got distracted with a Teams call.

Then I had a *well, duh *brainwave. I just needed to be stupid and write a series of joins. First just use SQL to count(*) and also the rows which had values for all columns. Then use that to knock out the columns in another table. Get smart about how I write the join, then that way I’m not manually writing down 200+ column names to knock out.

I didn’t have time to execute it, but my intuition would always go for a “dumb” solution like this because:

• KISS (Keep It Simple Stupid)
• Easier to debug
• Less likely to get lost along the way
• Easier for someone else to understand what I’ve done, and why I did it that way
• You can spot interesting stuff while doing the mid wrangling steps. You also get a feel for the table size. The keys you can join on. The column formats. At a certain point you just know when your table has duplicates because of something dumb you did, rather than needing to check.

I checked the log, the convoluted proc means method had hit a late stage error. I cut the code and left it fresh to review another day…

Intuition.

I’m not saying I’m better than an AI/LLM. Neither am I claiming we should abandon these tools. What I’m saying is it is incredibly hard to have an LLM plug into my intuition. To communicate something that I can’t even verbally communicate to myself. And that I wouldn’t have gotten that intuition if I hadn’t solved some gnarly data problems in my early Data Journalism career.

I don’t regret educating my intuition. I also can’t wait to educate it further.

---

Don’t mind me, just gonna plug my newsletter below:

This sign up might be intuitive

I can’t promise to educate your intuition, but I can promise some pretty slick data essays. Drop your best email below if you dare.

Sign-up form not appearing above? Pesky javascript blocker… click here and scroll to the bottom for a vanilla boring mailchimp form that works on all devices.

Footnotes

• 1

Yeah, that’s an affiliate link. Don’t like it. Don’t click it. But if you like the books, would be nice to support this piece of writing by buying from it. My book fund will greatly appreciate it.

• 2

Huge caveat the book was published in 2001, and a lot has changed in that time (Miley Cyrus was 9 and we didn’t yet know Britney Spears was about to have a lousy life). While the recently popular work of Amos Tversky & Daniel Kahneman has investigated how we make unconscious decisions (System 1 thinking), it focuses on the flaws in our reasoning, and how we fool ourselves. Other investigations into intuition focus on mastery and those who spend an ungodly amount of hours carrying out deliberate practice (10,000 hour rule). To my knowledge there is a gap in the middle ground: intuition that comes from everyday experience. Social intelligence in the workplace (and from school) is an example. There have been murmurs about the generational gaps in workplace social intelligence between Gen-Z and Millennials, and I’m sure there are some studies being carried out looking at Covid infants and Covid Graduates. But social intelligence is obvious. What is less obvious: the intuition you get from going to the office, from failure, from watching your peers work etc.

• 3

Used to be called “Statistical Analytics Software”. It is still used by large legacy institutions. Like…banks. And I work in one. https://www.sas.com/en_gb/software/base-sas.html

• 4

WHERE ... IS NULLor equivalent.

• 5

Enterprise approved version, because, again. Big bank. Consumer data protection. Ya get the gist.