I’m using ChatGPT for prototyping ideas. It’s kind of like this: I’m treating it like an intern, telling it "please write a script for me that does the following: …". It then gives me the script (an intern would do so after a couple days, ChatGPT does it in a few minutes), I can try it, sometimes it works, sometimes it doesn’t. If it doesn’t or I want something changed, I tell it (just as I would the intern), and it tries to do that.

Now, the funny thing is: sometimes my AI counterpart feels dumber than a few hours earlier. When earlier, it gives me a series of patches in a row, a couple hours later it forgets what the last version was and patches an ancient version instead, re-introducing bugs that it fixed before.

It’s like there would be different people I am talking to at different times, some more, some less competent.

Just today, it did something like that, some data retrieval from an API with very good, publicly available, documentation. It sent an ill-formed request and got an error.

Half a day earlier, it simply fixed that bug, and we moved on. Tonight, it didn’t just take the function from an older version, it wasn’t even able to check the documentation again or simply use that function from a recent version of the script. Instead, it proposed to define a fallback behaviour, which simply turns off the entire feature and breaks everything, and just issue a warning message in the output. Since the code was wrong, this would always have failed…

I pasted an example, working, example call to that API then with response, and suddenly it was able to fix it…

Does anyone else also have the impression of dealing with different shifts of counterparts, and it is pure luck if you find a competent one?