A couple of weeks ago at Eurosky I talked to some friendly Erlang hackers who wanted to get involved with AT development. The AT network is a big data-intensive distributed system, and it is the sort of thing the BEAM runtime is well-suited for. I know Chad Miller has been using Gleam for parts of Slices, and services like relays, jetstream, and the forthcoming tap tool could all be re-implemented using Erlang-y tools. But I think these tools, written in Go, are already in a pretty good place: they are efficient enough, are not too hard to operate (IMO), and have capacity to scale. hose.cam lists 9 full-network relays from 7 distinct parties.

I think the harder technical problem in the network today is patterns for big indexing: datastores for indexing billions of records. This is an area where there aren’t clear choices, and folks are feeling pain. There probably isn’t going to be one solution that works for everything, and more experiments and write-ups would be welcome, particularly from folks from outside the AT ecosystem who have a hammer they love (eg, a particular database) and are looking for a nail (a fun use case).

There definitely are folks doing great work in this direction already:

microcosm is a really impressive collection of generic AT services and APIs that work with all data in the network, is open source, and runs on affordable hardware

Bluesky itself switched from PostgreSQL to ScyllaDB for our appview, as described in this April 2024 deep dive. That was mostly driven by request volume, not indexing workload

futur.blue demonstrated a near-full-network Bluesky appview running PostgreSQL on a $200/month Hetzner instance

Blacksky has been building a full-network Bluesky-compatible appview, using a custom Rust indexing tool (wintermute)

Building alternative bluesky-compatible appviews is definitely a big motivating use case, but I think the need is broader than something that works for one project and codebase. At a minimum, it should be possible to develop new product features which would require additional data types and indices, like Blacksky is planning with community features. And we expect other projects and apps in the network to grow over time: we need to be ready for non-bsky record types with millions and billions of records, which will have their own unique indexing needs.

The sweet spot to me is: what systems and design patterns work well to index tens of billions of records on low-end bare metal servers? By "low-end" I mean cheap dedicated servers on the order of $100 to $600 a month: very expensive compared to a Raspberry Pi or basic VPS, but much cheaper than buying a $20k to $50k closet monster, and usually a lot cheaper than a dedicated/managed database server (eg, a big AWS RDS instance). This class of machine usually has tens of GB of RAM, 12+ vCPUs, and most importantly, several TBytes of fast directly attached NVMe storage. You can get good deals on this sort of hardware from OVH or Hetzner; don’t bother with cloud providers like AWS, GCP, or Azure. Ideally the datastore would support horizontal scalability for read load and availability, but still work well on a single instance for prototyping.

Some of the broad categories I can think of are:

getting regular PostgreSQL to scale better. I don’t mean sharding or Citus or alternative data wrappers (though those approaches are also fine), I just mean running regular modern PostgreSQL with TBytes of storage and indices on a single machine. This would mostly be about careful schema and query design. The big win here is that PostgreSQL "scales down" very well, lots of developers are familiar with it, the pitfalls are known, and there is a lot of good tooling available. As a side note, the current Bluesky appview codebase includes a PG backend, but it isn’t well optimized for current network scale.

NewSQL distributed database systems like Vitess, yugabyteDB, TiDB, ClickHouse, and many others. These are a bit more familiar to work with (still SQL), though might require distinct data layouts or have other quirks.

Columnar stores like Cassandra-compatibles, DuckDB, etc. Some of the NewSQL projects overlap with this category. The big strength is most likely compression efficiency, requiring much less disk space and enabling faster aggregation queries. The difficulties might be random write indexing performance (eg, from firehose), and cost/limitations on joins and secondary indices.

Key/Value stores like FoundationDB and ScyllaDB (which might overlap with other categories). These often have great performance and operations, but big learning curves around data layout, and don’t "scale down" well.

Hybrid, multi-modal, and "all others", like Elasticsearch.

Datastores are often branded or perceived as more transaction-oriented (OLTP) or analytics-oriented (OLAP), but can often work well enough for the opposite use case, especially if there is flexibility around performance or eventual consistency.

What would love to see emerge is a bunch of blog posts and trip reports talking about big AT data indexing attempts, and what the resource costs, bottlenecks, and pain points were. Maybe even a benchmark/leaderboard could emerge around how long it takes to backfill the full network and how much it costs (though this might be reductive and hard to do fair comparisons). I’m less interested in making a big list of hypothetical options, or "what about XYZ" questions: there are a bajillion ideas and options, we need real attempts for real use cases.