I’ve bumped into a couple of different Reddit discussions where people are sharing their spectacular/frustrating failures with AI agents. Both from the perspective of the “AI expert” and from the companies that hired them. I’m not so shocked at the former. Oil-snake salesmen have existed since forever. It is the latter that surprises me.

A few weeks ago, I published on LinkedIn an article where I shared my experiences with integrating AI at Trippz (https://lnkd.in/dSgZciGT). As we are now beginning another dev cycle, and I just realized that we are nearly completing one year (yes, 12 months) of this effort, I thought I would share a few more thoughts.

When I began working on the first iteration of our internal AI workflow, it took me around 2 weeks to get my first success. I was happy to see it work. We gave the AI a bunch of inputs, content, and parameters, and some minutes later, we had something valuable that our team could use.

Why, then, am I still working on this 12 months later?

Well, for one, we could say that AI is unpredictable. What works now may fail in the next run. We could also say that our inputs are so vastly different every time that one success means very little in terms of how reliable the whole system is. It took me 10 months to bring the failure rate to a solid zero. Today, I’m very confident that the system doesn’t fail with weird errors or unexpected AI behavior.

Secondly, it’s only when you are deep into the business and the user’s workflow that you can really understand how to make an impact. Those first 2 weeks could offload 2 hours of boring work from a teammate. But the next months helped make the many remaining hours even shorter and easier.

Then we can speak about improving the entire system itself. Just because something works doesn’t mean it’s well-built or efficient. I’ll give you real numbers: – It took me 1 year to bring down the runtime from ~10 minutes to ~60 seconds (avg) – It took me 1 year to bring down the cost of ONE run from ~4.00$ to ~0.50$ (avg) This is not only impactful from the point of view of usability and productivity, but it will also enable us to scale up the whole system faster and with more confidence. How was this achieved? This is where the other 50 weeks come in: it takes time! It takes time to know the business logic, and it takes time to understand the user’s pain points. It takes time to experiment. It takes time to come up with novel and creative strategies to solve problems (and no, ChatGPT is not going to do that for you)

So if you are a decision-maker in your company, and a snake-oil salesman is coming to sell you quick AI solutions, don’t be *that* kind of manager. At least get your technical team involved and trust them.

Unless, of course, what you are trying to do is piping emails to get an automated reply from ChatGPT. In that case, go ahead and deploy your new and shiny n8n template.