22 Dec 2025 by cthos 1779 words
So, I’ve been watching a bunch of folks whom otherwise have reasonable takes about LLMs who, under very specific circumstances, have extremely weird takes about how much an LLM is helping them do a thing, and their internal justifications for them. This isn’t about anyone specific; for each of these points, I’ve seen variations of these from more than one person in every case. But they’re happening enough that I want to talk about them. Don’t just take it from me, this post from @xgranade makes the point better than I can:
It is amazing how many people I see claiming that AI has one good use. No one seems to agree on what that one good use is, and there’s always a hell of a lot of goalpost shifting and special pleading involved. For some reas…
22 Dec 2025 by cthos 1779 words
So, I’ve been watching a bunch of folks whom otherwise have reasonable takes about LLMs who, under very specific circumstances, have extremely weird takes about how much an LLM is helping them do a thing, and their internal justifications for them. This isn’t about anyone specific; for each of these points, I’ve seen variations of these from more than one person in every case. But they’re happening enough that I want to talk about them. Don’t just take it from me, this post from @xgranade makes the point better than I can:
It is amazing how many people I see claiming that AI has one good use. No one seems to agree on what that one good use is, and there’s always a hell of a lot of goalpost shifting and special pleading involved. For some reason a lot of folks quite reasonably follow the arguments against AI, but then partition off the one thing as immune or exempt from having to worry about any of the ethical and practical problems. - @xgranade@wandering.shop
Let me start, though. I think each and every one of these things ignores the externalities of running an LLM: The wholesale scraping of data frequently without consent (and the associated pushing of costs onto the site being scraped), the massive build-out of data centers that will never be used, a global chip shortage because the AI industry can just do that, the proliferation of slop and disinformation, and the trend of people experiencing psychosis when interacting with chatbots.
Okay, with that out of the way, let’s dig in.
These are paraphrased. They are not direct quotes. If I use quotes, they are scare quotes to separate them from the rest of the sentence.
LLMs are great for Prototypes - I’d never have the time or ability to do this. It’d take me forever, or I’d never get it done.
Okay, so this one I’ve seen a whole lot lately. Namely, the claim that busy engineers are able to rapidly prototype something and get real time feedback on the prototype. This is usually associated with a sense that the engineer in question is "not a {insert type of} engineer" and so they’d never be able to complete such a task.
There are a bunch of problems with this approach, but I want to start with the claim that these folks would never be able to do something. That’s patently untrue. If you’re a software developer you absolutely have the capability to learn how something works and build out a prototype. The LLM’s statistically average output is going to produce code that is prevalent. The most prevalent front end code being fed through these things is React. React has some very silly contrivances in it, but it’s not an insurmountable mountain to learn the basics of React to roll out a prototype.
For code that is more difficult to learn, an LLM is going to have a harder time than you would, because the amount of samples in its training set are probably going to be a lot smaller.
The second problem with this attitude is that if you acknowledge that the code an LLM generates is substandard, and you want to treat it as throwaway.... it is never throwaway. Prototypes tend to make it, unmodified, into production. And if your usual engineers are busy, no one is going to substantively review this code.
Another issue is that the human involved in this process is not going to learn much from this process. Having the LLM write the code, and then try to make sense of the generated code is a failing proposition. The LLM does not have cognition. It produces code that is statistically likely. That’s going to be a twisted mirror of the code it was trained on. Will it execute? Maybe! It might even produce code that executes on the first go. That’s how slot machines work.
Also, this is why scaffolds and quickstarts exist?
I have the LLM do initial code review for me and point out things that are wrong
Permalink to “I have the LLM do initial code review for me and point out things that are wrong”
This one is the least offensive to me on this list, but it’s still super annoying that I have to, for example, review a bunch of suggestions that Copilot makes on a GitHub PR that are either subtly wrong, or correct enough that accepting its suggestion won’t break anything but it’s also not doing anything of value (for example, making a comment more verbose).
I’m not convinced that this actually saves more time than it wastes. I’ve not had it find a single bug that would have caused a major problem had it made it into production, and it has (had its suggestions made it through) introduced bugs that would cause problems.
Perhaps I’m also being a bit petty, I can discuss a problem with a human and get to the point where we understand where there’s a misunderstanding on either side - the LLM doesn’t do that, it’ll just agree with whatever you type to it, whether or not you’re actually wrong. Because it’s statistics.
It’s worth pointing out that you, writing good unit tests and utilizing static code analysis (linters, anyone?) should be able to catch the same amount of stuff that an LLM would, deterministically.
It’s great for generating ideas!
Permalink to “It’s great for generating ideas!”
Oh gods. Okay. So. I’m also a writer (fairly novice, granted), and I’ve found that there’s nothing about typing at an LLM that actually generates quality ideas. It just spits out mild rehashes of old ideas and frequently the same variant of an idea a bunch of times in a row.
Why would I use electricity for this? I can grab a coffee, stare out the window, or talk to an inanimate object on my desk and generate a ton of stupid ideas that I can then iterate on, and I’ve found that works out way better than running a GPU to spit out "ideas".
I encourage you to give this a try. Find somewhere quiet, stare out of the window, and let your mind wander. Ideas will come.
LLMs are awesome at summarizing! I’m a busy professional and there’s a lot of stuff for me to read!
Nope. LLMs do not summarize, they shorten. They also frequently introduce inaccuracies or make up quotes when they’re doing it. Often it’ll also remove critical information, or emphasize the wrong thing. Overall, you cannot trust even the most advanced LLMs to accurately summarize a longer piece of text for you.
So what do you have to do if you want to be sure you have an accurate understanding of a longer text? You have to read it yourself. Which defeats the purpose of having the LLM shorten the text for you in the first place.
If you don’t care about the accuracy of the summary...why are you reading it?
LLMs are great as a generalized search engine!
Permalink to “LLMs are great as a generalized search engine!”
So, this one is weird because it kinda depends on what you mean by "search engine". An LLM by itself isn’t a very good search engine. It’s maybe an okay reverse thesaurus or dictionary, but if you’re looking for something with up-to-date information you’re probably taking about something like Perplexity or the like.
Note, you can ask an LLM to generate "citations" and "link to your source" but it’s reasonably likely to hallucinate those.
In the case of Perplexity and Google LLM search, what’s actually happening behind the hood is a bit...insidious. They’re using Retrieval Augmented Generation, namely, doing a web search first, and then feeding the first couple of results into the LLM to have it "summarize" (see previous comment) the results, and using a couple of techniques to inject the source reference into the "summary". Couple of major problems with this:
- It’s only going to use the first couple of results for its summary, lest it risk overrunning its context window.
- It removes the results from their original context, so you can’t be sure the summary doesn’t include a shitpost from Reddit.
- "Hallucinations" still can crop up in those summaries.
Is this better than Google search has become? Maybe! Google has gotten pretty dang bad. I’ve resorted to running SearXNG on my own infra to get less bad results and no summaries.
Relatedly, we’re seeing more and more hallucinated citations appearing in places they shouldn’t so, uh.
This is just placeholder dialog, it’ll be removed before the final product is out
Permalink to “This is just placeholder dialog, it’ll be removed before the final product is out”
This one is a two-parter. Namely, the using of an LLM for "filler" text, and the use of a text-to-speech model to have verbal dialog - most commonly in games.
The first is a problem for even if you intend to remove it, bogus "filler" text will blend into the background. It will leave its mark on the final product. The odds that you’ll remember to remove it decrease, when you could just leave the prompt in ("Emotional scene where Santa admits he made a pact with a devil for immortality", e.g.) and it’d be a lot clearer what needs to be rewritten. By not doing that, you’re creating background radiation. Mediocre stuff that you may just miss when you’re doing your next dialog pass.
Like a lot of the use cases I see for generative AI, it boils down to "just send me your prompt" rather than having the token spewer spit out tokens.
The latter is just... Like. If you need filler audio so you can "hear" the scene, just have someone around you read it out loud? Record it yourself? The same thing as before applies, if you don’t intend to keep it in, there are other ways to get that out that do not require much more effort. You’ll get more emotional nuance from a person who understands the scene, rather than the flat garbage that Amazon tried to pull.
That’s all I’ve got energy for at the moment, but there are definitely more of these floating around out there, so I’ll probably update this post later to add more, or make a follow-up. Haven’t decided. I’m just so tired.