First off, I’m an engineer that did a lot of work on scaling, and in recent years, open source. I published https://github.com/titpetric/etl months if not years before I picked up AI for the first time. I wrote a lot of code in various runtimes and business domains but use Go exclusively for many years now. For anything.

My recent obsession with AI (in a measured way, unlike my obsession with coffee), lead me down a chain of writing supportive tooling like a template engine that works with hot-loading, follows Vue syntax and let’s me do back end templating in a familiar style. Convenience is king, and for me, convenience only means the go runtime, none of this node/npm ecosystem chaos, no scaling issues, and no needing to patch language syntax. If I had written it a few years ago, I wouldn’t have fs.FS, or generics, or iterators, and really the only concerns go code is left with is optimizing software design to new abstractions.

I implemented etl as a simple CLI, which grew into a server, where you would with a yaml configuration define a full API for your service, directly implementing the api with SQL. I added sqlite, mysql, and postgres, considering user choice. It enabled creating REST style APIs like /api/users/{id}, and spitting out the SELECT statement result as the response json.

Now I realize this is where AI accelerated me somewhat ; I added an additional handler that is able to invoke the API endpoint returning json and feed it to the template, which can now be defined in the yaml config. Additionally I asked for a rate limiter, defined the data models, extended tests, along my overlay of design, architectural and testing concerns. Software is never perfect, but iterative.

Why do you care? Well, here is where it gets interesting. Using sqlite I can simplify my database management (no connection pools and other limitations), meaning I’m only limited by disk and I can set very predictable quotas.

50mb per would partition a 500gb so many times that a single server could handle thousands of users.

Using the .yml gives me a sandboxed but unified execution environment, memory wise it can live even on a low memory instance serving tens of thousands of requests per second.

So my main problem is, is the SQL in yaml approach expressive enough? Can it build large composable systems? For this, I need testers. I can build and design apps on my own, and use this in process, sure, but the true step forward is someone that wants to do something with data. The more someone’s I have, I can see how this scales with real use with various applications that you could choose to model with SQL.

What’s in it for you? I can partition some cloud resources that give you an always on API that’s driven by an sqlite database. You could have a dashboard that queries data from sqlite, renders to JSON or HTML, has cached responses and configurable rate limits.

What’s in it for me? I obviously don’t care about market validation, more about the process. In the past I’ve relied too much on php, node and even go to implement APIs, always sort of falling back on the same operational problems. That being said, a PaaS that’s cost effective to run for this setup mainly needs to account for data durability, the traffic itself is an "add more nodes" problem. Since it’s a shared runtime environment the number of running processes per server is 1, per any amount of users. I love it.

It’s kind of hosting, but it’s really lightweight, so don’t think there’s a cutoff ; 10gb of storage is 50mb x 200, so lets make it anywhere from 200-500 users. Not to be bill gates and say 50mb is enough for everyone, but I can bump the quota, the only thing I can’t support is endless growth, at which point we have a discussion.

The limiting factor is cpu. Cpu I suspect will be most used if you’re doing statistics or querying the service without caching or limits. As you can configure those, not much concern it left.

Anyone willing to help in any way is welcome to reach out to me@titpetric.com, put ETL in the subject, like "I’d like an ETL server shard".

Don’t expect an immediate response, but if you include some detail as to what you’d use it for, it may get your onboarding fast tracked. Of course, you can build the docker image and start in your homelab, file any github issues, PRs.

Thank you for consideration. I’m really discovering the use cases and limitations here, and figuring that out is a people problem. I need people to poke holes in the design, point out edge cases.

Disclaimer: the project is OSS, the server is self hosting, written in Go, and I’d like to share this much as Linus Torvalds would (free as in beer).

I would add an AI policy, but other than "I trust it as far as I can throw it" the nuances of working with AI in my case only equate to my own dominion over it’s output, it’s not a shortcut for thinking things through. We both make errors. I lean into linting and detailed testing with test fixtures to mitigate regession, as I would for my own code. I favour composition. I haven’t seen a policy on AI use much as I haven’t seen policies for working with other devs, but I imagine they would be about the same. I’m having the same corrective reviews either way, that’s what you get from the average distribution of model training data.