Try Gemini 2.5

Our most intelligent model is now available on Vertex AI

Try now

In today’s fast-paced, data-driven landscape, the ability to process, analyze, and act on vast amounts of data in real time is paramount. For businesses aiming to deliver personalized customer experiences and optimize operations, the choice of database technology is a critical decision.

At Zeotap — a leading Customer Data Platform (CDP) — we empower enterprises to unify their data from disparate sources to build a comprehensive, unified view of their customers. This enables businesses to activate data across various channels for marketing, customer support, and analytics. Zeotap handles more than 10 billion new data points a day from more than 500 data sources across our clients, while orchestrating through more than 2000 workflows — one-third of those in real time with milliseconds latency. To meet stringent SLAs for data freshness and end-to-end latencies, performance is crucial.

However, as Zeotap grew, our ScyllaDB-based infrastructure faced scaling challenges, especially as the business needed to evolve towards real-time use cases and increasingly spiky workloads. We needed a more flexible, performant, cost-effective, and operationally efficient solution, which led us to Bigtable, a low-latency, NoSQL database service from Google Cloud for machine learning, operational analytics, and high-throughput applications. The migration resulted in significant benefits, including a 46% reduction in Total Cost of Ownership (TCO).

The challenge of scaling real-time analytics

Zeotap’s platform demands a database capable of handling a high write throughput of over 300,000 writes per second and nearly triple that in reads during peaks.

As our platform evolved, the initial architecture presented several hurdles:

Scalability limitations: We initially self-managed ScyllaDB, on-prem, and later on in the cloud. We use Spark and BigQuery for analytical batch processing, but managing these different tools and pipelines across our own environment and customer environments reached a peak where scaling became increasingly harder.

Operational overhead: Managing and scaling our previous database infrastructure required significant operational effort. We had to run scripts in the background to add nodes when resource alerts came up and had to map hardware to different kinds of workloads.

Deployment complexity: Embedding third-party technology in our stack complicated deployment. The commercial procurement process was also cumbersome.

Cost predictability: Ensuring predictable costs for us and our clients was a growing concern as our business grew.

These challenges drove us to re-evaluate our data infrastructure and seek a cloud-native solution that could meet our streaming first, “zero-touch” ops philosophy, while supporting our demanding OLAP and OLTP workloads.

Why Bigtable? Performance, scalability, and efficiency

Zeotap’s decision to migrate to Bigtable was driven by four key requirements:

Operational simplicity: Moving from ScyllaDB cluster to Bigtable meant eliminating a significant operational burden and achieving “zero-touch ops”. Bigtable abstracts away hardware mapping and node management. This eliminates the need for maintenance windows and helps ensure data rebalancing.

Performance: Zeotap needed predictable performance, even in the face of regularly unpredictable workloads to meet our stringent SLAs. Bigtable’s ability to deliver low latencies for both reads and writes at scale was crucial — especially with spiky traffic patterns.

Efficient scalability: Managing ScyllaDB cluster scaling, rebalancing, and hotspots was operationally intensive. Zeotap handles very spiky and bursty workloads at times exceeding 300,000 writes per second. Bigtable disaggregates compute and storage, allowing for rapid scaling (further enhanced by autoscaling), which automatically adjusts cluster size in response to demand. This lead to more cost efficiency and helped eliminate idle resources.

Total cost of ownership (TCO): A significant driver of this migration was the need for cost efficiency and predictability. By moving from ScyllaDB to Bigtable, we achieved a significant 46% reduction in our TCO. This stems from Bigtable’s efficient storage and the ability to combine use cases, such as using Bigtable as a hot store and BigQuery as a warm store.

• Tight integration: Bigtable’s integration with other Google Cloud services, particularly BigQuery, was a major advantage in reducing operational overhead. Features like reverse ETL directly into Bigtable greatly simplifies data pipelines and reduces Zeotap’s operational footprint by 20%.