Written by Katie Shannon

Background

At Reddit, we have four Core Models that power pretty much all use cases: Comments, Accounts, Posts and Subreddits. These four models were being served out of a legacy Python service, with ownership split across different teams. By 2024, the legacy Python service had a history of reliability and performance issues. Ownership and maintenance of this service had become more cumbersome for all involved teams. Due to this, we decided to move forward into modern and domain-specific Go microservices.

In the second half of 2024, we moved forward with fully migrating the comment model first. Redditors love to share their opinions in comments, so naturally the comment model is our largest and highest write throughput model, making it a compelling candidate for our first migration.

How?

Migrating read endpoints is typically well understood and the solution is straightforward; we utilize tap compare testing. Tap compare is a way to ensure that a new endpoint is returning the same response as the old endpoint without risking user impact. We simply direct a small amount of traffic to the new endpoint, we get the response generated by the new endpoint, then call the old endpoint (from the new endpoint), and compare and log the responses. We still return the response from the old endpoint to the user to ensure no user impact, and have logs captured if the new endpoint would have returned something different. Easy AND safe!

On the other hand, write endpoints are a much riskier migration.

Why? Firstly, write endpoints almost always require writing data to datastores (caches, databases, etc). We have a few comment datastores to worry about, and we also generate CDC events when anything changes on any core model. We provide a 100% guarantee of delivery of these change events, which other critical services at Reddit consume, so we want to ensure there is no gap, delays or issues in our eventing generation. Essentially, instead of just returning some comment data like in our read migration, our comments infrastructure has three distinct data stores that are written to that factor into the migration:

Postgres – backend datastore which holds all of the comment data

Memcached – our caching layer

Redis – the event store used to fire off CDC Events

If we simply tap compare a write migration without any special considerations for the data stores, we could get into a state where the new implementation is writing invalid data, which fails to be read by the old implementation. To safely migrate Reddit’s most critical data, we could not rely on validating tap compare differences within our production data stores.

Due to unique key restrictions on comment ids, duplicate writing to our data store is impossible. So, how does one validate a write to our data storage from two implementations without committing the same data twice? Thus, in order to properly test our new write endpoints, we set up three new sister datastores to be only used for tap compare testing, and only written to by our new Go microservice endpoints. That way, we could compare the data in our production data stores written by the old endpoint with the data in these sister data stores without the risk of the new endpoint corrupting or overwriting the production data stores.

To verify these sister writes:

We directed a small percentage of traffic to the Go microservice 1.

The Go microservice would call the legacy Python service to perform the production write 1.

The Go microservice would then perform its own write to the sister data stores, completely isolated from the production data This diagram shows the dual write process for comments during tap comparison.

After all writes were done, we had to verify them. We read from the three production data stores that the legacy Python service wrote to, and compared them to what we wrote to the three sister data stores in the Go microservice.

Additionally, to combat some serialization issues we ran into early in the migration process, where Python services couldn’t deserialize data written by Go services, we verified all the tap comparisons in comment CDC event consumers in the legacy Python service.

This diagram shows the verification process of the tap compare logs that takes place after the dual write.

In summary, we migrated 3 writes endpoints, that each wrote to 3 different datastores, and verified that data across 2 different services, resulting in 18 different tap compares running that required extra time to validate and fix.

Outcome and Improvements

We are excited to say that after a seamless migration, with no disruption to Reddit users, all comment endpoints are now being served out of our new Golang microservice. This marks a significant milestone as comments are now the first core model fully served outside of our legacy monolithic system!

The main goal of this project was to get the critical comments read/write paths off the legacy Python service to a modern Go microservice while maintaining performance and availability parity. However, the migration from Python to Go yielded a happy side effect where we ended up halving the latency for the three write endpoints that were migrated. You can see this in these p99 graphs, (old legacy Python service endpoints are green, and new endpoints in the new Go microservice are yellow).

Create Comment Endpoint

This graph shows the 99th percentile latency for the endpoint called when creating new comments. The green represents calls handled by the Python monolith, whereas the yellow represents calls from the Go microservice.

Update Comment Endpoint

This graph shows the 99th percentile latency for the endpoint called when updating comments. The green represents calls handled by the Python monolith, whereas the yellow represents calls from the Go microservice.

Increment Comment Properties Endpoint

This graph shows the 99th percentile latency for the endpoint called when incrementing properties of a comment, such as upvoting. The green represents calls handled by the Python monolith, whereas the yellow represents calls from the Go microservice.

These graphs are capped at a .1 x axis (100ms) so the difference is visible, but the legacy Python service occasionally had very large latency spikes up to 15s.

What We Learned

The comment writes migration, while successful, provided valuable insights for future core model migrations. We came across a few interesting issues.

Differences in Go vs. Python

Migrating endpoints between two languages is inherently more difficult than, say, a Python to Python migration. Understanding the differences in the languages and how to generate the same responses at the Thrift and GRPC level was an expected difficulty of the project. What was unexpected was the underlying differences in how Go and Python communicate with the database layer. Python uses an ORM to make querying and writing to our Postgres store a bit simpler. We don’t use an ORM for our Golang services at Reddit, and some unknown underlying optimizations on Python’s ORM resulted in some database pressure when we started ramping up our new Go endpoint. Luckily, we caught on early and were able to optimize our queries in Go. Moving forward with future migrations, we’ve ensured to monitor our database queries and resource utilization.

Race Conditions on Comment Updates

Tap compare was a great tool to ensure we didn’t introduce differences with the new endpoint. However, we were getting “false mismatches” in our tap compare logic. We spent a long time trying to understand these differences, and it ended up being because of a race condition.

Let’s say we’re comparing an update comment call which updates the comment body text to “hello”. This update call gets routed to the new Go service. The Go service updates the comment in the sister data stores, then calls the Python service to handle the real update. It then compares what the Python service wrote to the production database, and what Go wrote to the sister database. However, the production database’s comment body is now “hello again”. This caused a mismatch in our tap compare logs which didn’t make much sense! We realized this was because the comment that was updated had been updated again in the milliseconds it took to call the Python service and make the database calls.

This made things complicated when trying to ensure that there were no differences between the old and the new endpoint. Was there a difference because of a bug in the implementation between the old and new endpoint, or was it simply an unluckily timed race condition? Moving forward, we will be versioning our database updates to ensure we’re only comparing relevant model updates.

Tests

A lot of this migration was spent manually poring over tap compare logs in production. Moving forward with future core model migrations, we’ve decided to invest more time having more comprehensive local testing before moving forward with tap compares in hopes that we’ll catch more differences in endpoints and conversions early on. This isn’t to say there weren’t extensive tests in place for the comments migration, but we’ll be taking it to an entirely new level for our next migration.

Each comment is composed of many internal metadata fields to represent different states a comment can be in – resulting in thousands of possible combinations in the way a comment can be represented. We had local testing covering common comment use cases, but relied on our tap compare logs to surface differences in niche edge cases. With future core model migrations, we plan to delve into these edge cases by using real production data to inform our local tests, before even starting to tap compare in production.

What’s Next?

The goal of Reddit’s infrastructure organization is to deliver reliability and performance with a modern tech stack, and that involves completely getting rid of our legacy Python monoliths. As of today, two of our four core models (Comments and Accounts) have been fully migrated from our Python monolith and in progress are the migrations for Posts and Subreddits. Soon, all core models will be modernized to ensure your r/AmItheAsshole judgements and cute cat pictures are delivered more reliably and faster!