In this iteration, we also attempted to add another layer of data gathering by capturing network traffic. The idea was to reverse-engineer the backend API by analyzing the JSON payloads passing between the frontend and backend.

While we eventually succeeded in generating a Swagger document of the API, this setup was much more difficult to figure out than the change data capture.

We also concluded that for this specific scenario — where the API is an internal implementation detail, not a public contract — the information about the backend API didn’t really give us anything new that we didn’t already know about the data structures.

Validation through rebuilding

The ultimate test was to use our AI-generated artifacts to create a new application. Our focus in this experiment wasn’t forward engineering, so we didn’t want to spend too much time on this step.

However, we still needed to see if the specification was of any use; so, we decided to feed it into one of the AI rapid application generator tools.

First we fed our detailed specification — the collection of Markdown files, screenshots and database logic descriptions — into a series of prompts that generated epics, user stories and an order of implementation. Then, we fed the first set of stories into Replit.

Some observations from this validation phase:

**Prototyping as a validator. **Watching the AI build the prototype was a very effective way to validate and improve the spec. When the prototype deviated from the original, it was sometimes easier to spot a flaw or ambiguity in our requirements document than it was when reading the specification.

AI struggles in planning. LLMs often struggle to create a logical, incremental implementation plan; that was the case here. Its first instinct was to build the entire, complex pipeline view in one go, rather than starting with the simplest elements and building up. A good AI workflow and human oversight is important to guide the build strategy in a real-world scenario.

**No demonstration of query fidelity. **Unfortunately, Replit in this case ignored our inputs about the database schema and the exact queries we wanted done. As we didn’t want to spend too much time with forward engineering, this meant we couldn’t fully show that the new application could be pointed at the existing database. However, Generative AI is usually quite good at picking up on specific examples and schemas and using them in the code; it’s not much of a leap to think that the specific database queries we collected could be reproduced by a coding agent with dedicated prompting.