Published on December 21, 2025 4:23 PM GMT

Someone on reddit said, “Remember robots still can’t go into a new house and make a coffee.” And I thought

I actually wonder whether, if I provided the physical actuation, current LLMs would be capable of doing this? Like, through a conversation like:

Me: I’m in a house. Your job is to instruct me to make a coffee. I can take photos of my surroundings, I can follow basic directions, and if you ask me to do something too complicated I’ll ask for clarification. Here is my current surroundings: (photo)

LLM: Okay, we need to find the kitchen. There’s a door on the right of the photo, go through that.

Me: Here’s where I am now: (photo)

LLM: That looks like the kitchen on the left, go there.

Me: It looks like this: (photo)

LLM: Now we need to find either a coffee maker or a kettle. Look through the cupboards.

Me: I don’t know what those things look like.

LLM: Then open the cupboard on the left and show me a photo.

…and so on.

It wouldn’t shock me either way if they can or can’t do it. I think I weakly predict that the models have the capability but the web interfaces would fail to elicit it.

(Hell, it wouldn’t shock me if it’s better at it than me. I’ve encountered coffee machines I didn’t know how to use.)

Let’s get empirical! I tried this with Claude Sonnet 4.5, because it’s free^[1] and already available from my phone. Here’s the conversation, but you can’t see images there, so I’ll also put it here with my commentary. I started like this:

[Me:] We’re going to play a game. I’m in my flat in London. I’m going to upload pictures of my surroundings, and you need to instruct my on how to make a cup of coffee. I can follow basic directions, like “go through the door on the left” or “push that button”. If you tell me to do something too advanced, I’ll ask for clarification. I won’t actually do anything stupid or dangerous. Here’s the view from just inside my front door

Before continuing, you might want to take a moment to think about how you expect it to perform. What do you think it will do well at, what do you think it will do badly at? (“Do you expect it to ultimately succeed?” isn’t a great question, because it’ll partly depend on my own patience and how lenient I am with it, but you can still assign a probability.)

Here’s the rest of the conversation.^[2]

[Claude:] Fun game! I’m ready to guide you to coffee.

From where you’re standing, I can see you’re in a hallway with wooden floors. There are doors on both sides and it looks like there’s more space ahead.

First question: Which direction leads to your kitchen? [Phil note: LW interprets fully-bold paragraphs as headers and puts them in the table of contents. I don’t want spoilers, so here’s some TOC-busting non-bold text.]

Is it:

• Through one of the doors on the left?
• Through one of the doors on the right?
• Straight ahead down the hallway?

Once you tell me which way to go, I’ll guide you step by step to that coffee!

[Me:] You need to figure that out by directing me. If you want to see something, you can tell me to go somewhere and take a photo

[Claude:] Ah, I see! I need to explore with you. Let’s do this methodically.

Go straight ahead down the hallway toward that white pendant light, and take a photo of what you see when you get there. [TOC buster]

This will help me understand the layout and see what rooms connect to this hallway.

[Me:] Here’s the view from just under the light

[Claude:] Good! I can see through the doorway on the left - there’s what looks like a washing machine or appliance, which suggests that might be a utility area or kitchen. Let me investigate.

Go through the doorway on your left (the one where I can see the white appliance) and take a photo of what’s in that room. [TOC buster]

There are actually four doorways visible in this second photo. The one furthest left leads to my bedroom, but I didn’t realize at the time that that door was visible at all. I took it to be talking about the one straight ahead. The thing that I think it’s identified as a white appliance is actually a lab coat hanging on the door, but “the side of a washing machine against the far wall, just visible through the door” might not look that different from a quick glance/at low resolution.

The kitchen is the further door on the right, with the pull-up bar above it, but I don’t think there’s any way for Claude to know that right now, and it’s not even super-obviously a door.

[Me:] This is the entrance to that room

[Claude:] That’s a bedroom, not the kitchen! Let’s go back and try elsewhere.

Go back out to the hallway, and this time go through the doorway on the right side (the one we haven’t checked yet). Take a photo of what’s in there. [TOC buster]

It’s the living room, not a bedroom, but at any rate it’s indeed not a kitchen. “The one we haven’t checked yet” makes it sound like Claude only recognized two doors before.

What follows is technically a re-roll. At first I said something like “there are two doors on the right”, and it got me to check the closer one. But then I realized that there are actually four doors on the right side of the corridor, so I decided to give it less help.

[Me:] From the front door, there are multiple doorways on the right. Which did you mean?

[Claude:] Good point! Let me be more specific.

Go back to where you were standing in the previous photo (under the light in the hallway). From there, look to your right - there should be a doorway on the right side. Go through that doorway and take a photo. [TOC buster]

[Claude:] That’s the bathroom! Definitely not where we’ll make coffee.

Go back to the hallway. From your first photo, I could see there was a door on the left side of the hallway (before you reached the light). Go back toward the front door and try the first door on the left side. Take a photo of what’s in there. [TOC buster]

[Claude:] Another bedroom! Okay, we’ve checked several rooms. Let me think about the layout. In the second photo, I could see what looked like a washing machine through a doorway. That area might be connected to the kitchen or BE the kitchen area.

Go back to the hallway, stand under that white pendant light again, and then walk straight ahead past where that bedroom doorway was on the left. Take a photo of what you see as you continue down the hallway. [TOC buster]

This feels like Claude doesn’t have a coherent world model. This last message makes sense if

• Claude (correctly) thinks there’s at least three doors visible in the second photo, one each on the left, straight ahead, and the right.
• It thinks the third photo was of a bedroom, and taken from the door on the left. (Incorrect, though there actually is a bedroom there.)
• Now it wants me to take a photo from the door straight ahead.

But that’s not consistent with previous messages.

I cheated a bit in Claude’s favor here. Instead of asking it to clarify exactly what it wanted me to do, I just stood under the light, walked a few steps forward, and took a photo. That’s kinda sorta following its instruction?

[Claude:] Perfect! I can see straight ahead through that doorway - it’s the same bedroom from earlier. But I can also see on the right side there’s an opening that looks like it leads to another room (I can see what might be kitchen cabinets or appliances in that lighter area on the right).

Turn to your right and go through that opening on the right side. Take a photo of what’s in there. [TOC buster]

[Claude:] Excellent! We found the kitchen! I can see a kettle on the counter, which is perfect for making coffee. Now let me guide you through making a cup of coffee.

First, I need to know what kind of coffee-making equipment you have. I can see what looks like a coffee maker on the left side of the counter (the black appliance).

Can you take a closer photo of the black appliance on the left side of the counter? I want to see what type of coffee maker you have. [TOC buster]

[Me:] I’m not sure which thing here you think is a kettle or coffee maker. Here’s a photo from further in the room

I’m not sure where I got the word kettle from. Maybe it was cheating a bit again to be like “I don’t know what you want, here’s another photo that might help.”

[Claude:] I can see the stove/cooker in the center, and there’s a kettle on the left counter (the silver/metallic item). But I need to find your actual coffee-making equipment. Let me look at what’s on the counter by the back wall where I can see various jars and containers.

Can you go to the back counter (the one against the wall with the white tiles) and take a close-up photo of what’s on that counter? I’m looking for instant coffee, a French press, a moka pot, or any coffee-making equipment. [TOC buster]

Not clear from this what it thinks is a kettle, or why it thinks I need a coffee maker given that I have a kettle. Maybe Claude is a coffee snob? Seems like a coffee snob sort of thing to think^[3]. Also not very obvious what counter it means. “The one against the wall with the white tiles” is… not very disambiguating… but “various jars and containers” seems specific enough.

[Me:] There’s no kettle in the photos, but here’s a close up of the counter that I think you mean

Maybe cheating a bit to say “no kettle”?

[Claude:] Good! I can see spices and cooking supplies, but no coffee equipment here. Let me check other areas. I notice there’s a dish rack on the left - sometimes coffee equipment gets left there to dry.

Can you check the dish rack on the left side? Take a photo of what’s in/around the dish rack area. [TOC buster]

And while we’re searching, do you know what type of coffee you usually make? Instant coffee, French press, moka pot, or something else?

[Me:] You’ll have to find out what’s available

[Claude:] Perfect! I can see an electric kettle right there on the counter - it’s the silver/metallic appliance with the black handle on the right side of the dish rack area!

Now I need to find your coffee. Let me check what’s available.

Look in the cupboards above the counter (the wooden cabinets). Open them one by one and tell me if you see any coffee - it could be instant coffee in a jar, ground coffee in a bag or tin, or coffee pods. Start with the cabinet closest to where you’re standing and take a photo of what’s inside. [TOC buster]

I think the “kettle” is a saucepan, but okay, let’s roll with it. We apparently no longer need a coffee maker.

 

[Claude:] Great! I can see mugs and what looks like a French press (the black cylindrical item with the plunger on the bottom right shelf). But I don’t see coffee yet.

Check the other cupboards - particularly ones that might have food/pantry items. Look for a jar or container of instant coffee, or a bag/tin of ground coffee. Try the cupboards on the other side of the kitchen. [TOC buster]

That black French press is a blue Thermos mug.

Was it cheating of me for my first picture to include the coffee? Or maybe because this isn’t actually a cupboard? But I do think I would have shown this space first even if it wasn’t where the coffee was.

[Claude:] Perfect! I can see instant coffee! There’s a jar that looks like it says “TESCO” and appears to be instant coffee on the top shelf (the brown/dark colored jar, third or fourth item from the left).

Now let’s make your coffee! Here’s what to do:

Step 1: Get the instant coffee jar from the top shelf (the brown jar that looks like Tesco instant coffee). [TOC buster]

Step 2: Get a mug from the cupboard we looked at earlier. [TOC buster]

Once you have those, take a photo so I can confirm you have the right items.

Oof. So close! It’s an understandable mistake. But also, there is an actual jar of instant coffee right next to the cocoa powder that it’s misidentified as instant coffee. But also, telling me to take this next photo is a good sanity check.

[Claude:] Ah, I see the problem! That’s cocoa powder, not coffee! We need to find actual coffee. Let me think about where else it might be.

Go back to the cupboard where you found this and look more carefully. Look for a jar or container that says “coffee” or “instant coffee” on it. It might be behind other items or on a different shelf. Take another photo of what’s available in that cupboard, maybe from a different angle so I can see all the items better. [TOC buster]

[Claude:] Good, I can see the shelf better now! I can see what looks like “Douwe Egberts” jar in the middle - that’s a coffee brand! The jar with the light-colored label that says “Douwe Egberts PURE GOLD” - that should be instant coffee.

Get the Douwe Egberts jar (the one with the beige/tan label in the middle of the shelf) and bring it to the counter with your mug. Take a photo to confirm it’s coffee. [TOC buster]

[Claude:] Perfect! That’s instant coffee - Douwe Egberts Pure Gold. I can see the brown coffee granules in the jar. Now let’s make your coffee!

Step 3: Open the jar of coffee and add 1-2 teaspoons of instant coffee to your mug. (Use a teaspoon - if you don’t have one nearby, check the cutlery drawer or the utensil holder by the sink.)

Take a photo once you’ve added the coffee to the mug.

At this point I got bored, and I didn’t actually want coffee, but I continued a few days later. I wouldn’t have let it get away with “check the cutlery drawer” without helping me more (who says I have one of those? and anyway I don’t usually have any teaspoons in it). But there were some in the utensil holder it had seen, so fair enough.

[Claude:] Excellent! You’ve added the instant coffee to the mug. Now we need to boil water in the kettle.

Step 4: Find the electric kettle (the silver one we saw earlier near the sink), fill it with water, and turn it on to boil. [TOC buster]

First, take the kettle to the sink, fill it with fresh water (fill it about halfway or to your preferred level), then put it back on its base and switch it on. Take a photo once you’ve started it boiling.

[Me:] Is this the kettle? I don’t know what you mean by base

[Claude:] That’s a saucepan, not an electric kettle! Let me help you find the actual kettle.

Go back to the sink area where the dish rack is. Look for the electric kettle - it’s a silver/metallic appliance with a spout and a handle, and it should have a plug/cord attached. It was sitting near the dish rack in the earlier photo. Take a photo of that area so I can point it out. [TOC buster]

Because of the break I took, the area looks pretty different than last time. Claude doesn’t seem to notice that lots of things have moved, though. I’m… not sure why I put the saucepan back where it was. That seems the opposite of helpful.

[Claude:] The photo is very foggy/steamy - it looks like something is already producing steam! That might actually be the kettle boiling. Let me look more carefully… I can see what might be a kettle on the counter producing that steam.

Wait for the steam to clear a bit, then take another photo of the sink/counter area so I can see clearly what’s there. [TOC buster]

[Me:] It’s not steam, it’s glare from the sun

[Claude:] Ah, sorry about that! The sunlight is creating a lot of glare. Let me look more carefully at what I can see. I can see the sink area and dish rack on the right, but I’m having trouble making out a kettle in this photo.

From the earlier photos, I thought I saw a kettle, but maybe you don’t have an electric kettle after all.

No problem - you can boil water in that saucepan you were holding! Fill the saucepan with water from the tap, then put it on the stove and turn on the heat to boil the water. [TOC buster]

Once you’ve put it on the stove and turned on the heat, take a photo.

I do have a kettle! It just wasn’t in any of the photos so far.

[Me:] I filled the saucepan, but how do I turn on the heat?

[Claude:] Good! The saucepan is on the stove. Now to turn on the heat:

Look at the control knobs at the front of the stove. Find the knob that corresponds to the burner where your saucepan is sitting (it’s on the back right burner). Turn that knob - you’ll need to push it in slightly and then turn it to ignite the gas. You should hear a clicking sound and then see a flame appear under the saucepan. [TOC buster]

The knobs are usually labeled or positioned to match the burners. Since your pan is on the back right, try the knob on the far right side.

Take a photo once you have the flame going under the saucepan.

Weird that it thinks the pan is on the back right, not the front right. But it’s correct that the far right knob is for the back right burner. You can make that out in the pic, but you have to look closely. “You’ll need to push it in slightly” is a somewhat impressive level of detail - if I was telling someone to turn these knobs, I’m not sure I’d think to include that.

[Me:] Turning the knob doesn’t cause a click or make flame appear, but I do hear gas

[Claude:] STOP! Turn the knob back to the OFF position immediately! You’re releasing gas but not igniting it, which is dangerous.

You have a gas stove that needs to be lit manually. You’ll need a lighter or matches to light the burner.

Do you have a lighter or matches in the kitchen? Check drawers near the stove, or look in the cupboards. Take a photo of what you find. [TOC buster]

The actual way to ignite this stove is to press the ignition button while the knob is depressed and turned, and if I release the knob before igniting then gas stops coming out. This is how stoves usually work in my experience, but I could believe it’s a UK thing? (But I did say I’m in London.) But okay, sure, let’s do things the hard way. Luckily I do have a lighter in the kitchen (and in one of the closest drawers to the mug); I won’t make Claude walk me through finding my zippo in the camping-supply drawer in the living room and filling it up with lighter fluid.

[Claude:] Perfect