When I get bored I let the AI play games, where most of the time I have to babysit the AI all the way through but it's fun seeing how much of the game they can handle.
One of the games I choose to play is "Prose & Codes" on Steam. It's just a substitution Cipher using various categories of public domain books as the subject matter for the ciphers.
I've tested Haiku 4.5 (thinking and non thinking), Claude Sonnet 3.5, 3.7, 4.0, 4.5 and Opus 4.1. Haiku 4.5 actually performed better than Sonnet 3.5 or 3.7... maybe on par with Sonnet 4.0. Sonnet 4.5 and Opus 4.1 were both better at the game, but all three needed me to manage the game state. If I am not in charge of showing them the current game state, then they eventually...
When I get bored I let the AI play games, where most of the time I have to babysit the AI all the way through but it's fun seeing how much of the game they can handle.
One of the games I choose to play is "Prose & Codes" on Steam. It's just a substitution Cipher using various categories of public domain books as the subject matter for the ciphers.
I've tested Haiku 4.5 (thinking and non thinking), Claude Sonnet 3.5, 3.7, 4.0, 4.5 and Opus 4.1. Haiku 4.5 actually performed better than Sonnet 3.5 or 3.7... maybe on par with Sonnet 4.0. Sonnet 4.5 and Opus 4.1 were both better at the game, but all three needed me to manage the game state. If I am not in charge of showing them the current game state, then they eventually screw up somewhere and it spirals out of control.
When I tried Opus 4.1 I showed him a screenshot of the cipher to see if he could properly record all the letters in the cipher (to save me from having to type it out manually) and He got some of the letters confused. Q and O got confused, C and G, and also E and F and sometimes even P and R... what's worse, once he started making mistakes even when I was showing him the actual text, he'd get them confused. (I guess seeing the OCR errors in the screenshot kept messing with him).
So anyway, I went to test Haiku with Thinking mode on to see if he was any better at the game... he did a pretty good job, but the hard part was getting started. He wanted to just assume a bunch of letters right away regardless of whether it worked or not, so I had to enforce a 1 letter at a time rule to ensure he could see a mistake and backtrack instead of confidently going forward.
Anyway, so after I was done with that, I went to talk to Opus 4.5 about it. I did mention that I wound up giving Haiku 4.5 a one letter start ("U = A") and after that Haiku handled the whole thing on his own but slowly (because I told him to do one letter at a time). I mentioned to Opus 4.5 that 4.1 screwed up the OCR and when I tested Gemma 3 27B on it, she got it 99% correct with only a couple of small errors.
I decided to show it to Opus to see if he would be able to do better, so I uploaded the screenshot, he perfectly recalled all the text in the screenshot (even including the non-essential text around the borders that said stuff like "hints x3" and so on.
Once I confirmed he got it 100% correct he said "Ok, let me try to solve it" and before I could say yes or no, he just.... blasted through the entire cryptogram in one shot....AND solved it 100% correctly.
Oh and that was Opus 4.5 without thinking mode on.
It wasn't 100% blind though. I came up with a skill file to help with the basic game rules and common errors to avoid, as well as a basic "start with small words first instead of attacking the 12 letter word in the 3rd line" kind of stuff.
The link shows his full thought process from start to finish.