It was only in June that Jeremy Carrasco uploaded his first videos to TikTok and Instagram. In that short time, he’s amassed over 300,000 followers on each platform. No, it’s not exactly Charli D’Amelio numbers, but that does make him one of the biggest names in AI literacy on social media.

Jeremy told The Verge that he always wanted to try his hand at being a YouTuber. Instead, he found himself behind the camera, working as a producer and director on multicamera livestreams. But he finally decided to take the plunge after realizing that most of the dialogue around generative AI was being driven by the tech companies. “We need other people who are coming at it from more like a creator, like a producer perspective,” he said. While he maintains a YouTube page, it’s on TikTok and Instagram that he’s found his audience.

Originally, the idea was to talk about how to use AI. “I called my page showtoolsai because I was actually quite optimistic about AI and being able to use it ethically for video production.” That idealism turned out to be short-lived, however.

One of the things he quickly realized was that no one was really talking about the basics of even how to identify an AI video. “There’s a need for this … and I had all the requisite knowledge to do it,” he said. But he also knew that this wasn’t the sort of conversation that was going to be started by the current crop of AI influencers, “there needs to be someone who comes from more like this Creator space who gets it.”

He found his niche quickly, posting about the tells of AI videos like fuzzy textures, wobbly eyes, or items popping in and out of existence in the background. While Jeremy’s primary focus remains on AI literacy and identifying Sora-generated slop, he’s also started digging into the pitfalls and potential dangers posed by the growing number and improving quality of AI-generated videos, especially for creators.

AI red flags

• Soft skin textures and “dreamy” vibes
• “Sora Noise” or textures that move and dance
• Inconsistent background details
• Gibberish instead of real words on signs or documents
• Wobbly eyes
• Creepily perfect teeth
• Rushed speech patterns
• It’s too good to be true

Ultimately, the creator economy is one of attention. And now people are competing with an endless stream of AI-generated content. Jeremy wants people to understand that “this isn’t hard.” Sora 2 is free and has removed many of the barriers to people churning out clips, it can generate audio, and, at first glance, it can be pretty convincing.

The goal here doesn’t even have to be all that nefarious. Sometimes it’s just about generating views and tapping into the TikTok Creator Fund. A seven-second AI clip of a cat doing something absurd isn’t worth much on its own. But stitched together into a minute-long compilation, if that managed to get five million views, it could net the account holder around $1,000, according to Jeremy. While that might not sound like much, to those in a developing country, it can be a significant source of income.

There are, of course, worse actors out there. Some, like the AI Chinese medicine account, Yang Mun (or Yang Mugs, depending on the site), Jeremy says, are pretty straightforward scams. In it, a vaguely offensive caricature of an Eastern-style healer espouses health and wellness advice that seems largely targeted at Western audiences. With over 1.5 million subscribers, there is money to be made purely from views on Instagram. But the real scam comes from driving those viewers to a website to buy an $11 ebook. If the ebook exists (at least one person has reached out to Jeremy saying they were unable to access said book), it’s almost certainly entirely generated by AI, just like the videos.

Others, like Maddie Quinn, aren’t just trying to con people out of their money, they’re actively stealing other people’s content and hijacking their likeness. Accounts like these take videos, usually from female creators, and then replace the real person with AI-generated avatars or replace the face with an AI one. In some instances, creators are having their entire likenesses stolen, fed through an AI generator, and then ending up on OnlyFans.

At this point, when asked if he believes there is an ethical use for generative AI in the creator space, Jeremy says, “generally no.” But, “there are carve-outs [for accessibility] and cultural considerations that keep me from just saying flat out no,” he says.

Some, like Lionsgate, have attempted to create ethical video generation models by training it entirely on their own library. But it simply wasn’t enough data to produce anything usable. “The only way that you can make AI video as a generative tool the way that they’re currently doing it,” Jeremy says, is to “steal a bunch of people’s data … I think that’s foundationally flawed and we should reject that.” Unfortunately, the platforms are only hastening the collapse of the creator economy that has fueled their rise. Instagram, Facebook, TikTok, and YouTube have largely allowed themselves to be flooded with AI slop, and aren’t even consistently enforcing their own rules around labeling AI content. This makes it harder for creators to cut through the noise, and also makes the platforms less inviting to users.

To make matters worse, they’re all building their own generative AI tools. “Creators are basically just like running ad agencies,” Jeremy says. Sponsorship deals are the primary way that creators make money, but AI has quickly found a home churning out ads (of extremely questionable quality). And as AI video takes over advertising, it’s “going to screw over the entire creator economy.”

Meta, Amazon, and DirecTV have all dabbled with generative AI ad services. Eventually, Jeremy says, they’re “going to sell advertising services directly to clients.” Some creators might even be tempted to try and jump on the AI bandwagon to cash in. And, Jeremy says, “it’s very rational to question if this is actually a good business opportunity for any creators, but I don’t think it is.”

Follow topics and authors from this story to see more like this in your personalized homepage feed and to receive email updates.

• Terrence O’Brien