Developers don’t care about Kubernetes clusters

opinion

Nov 6, 202510 mins

If you look at the Cloud Native Computing Foundation landscape it might seem cloud developers are a lucky bunch. There seems to be an existing tool for literally every part of the software development life cycle. This means that developers can focus on what they want (i.e., creating features) while everything else (e.g., continuous integration and deployment) is already in place. Right?

Not so fast. The CNCF landscape tells only part of the story. If you look at the cloud tools available, you might think that everything is covered and we actually have more tools than needed.

The problem, however, is that the cloud ecosystem right now has the wrong focus. Most of the tools available are destined for administrators and operators instead of feature developers. This creates a paradox where the more tools your organization adopts, the less happy are your developers. Can we avoid this?

Looking beyond the clusters

It was only natural that the first cloud tools would be about creating infrastructure. After all, you need a place to run your application, in order to offer value to your end users. The clear winner in the cloud ecosystem is Kubernetes, and many tools revolve around it. Most of these tools only deal with the cluster itself. You can find great tools today that

• Create Kubernetes clusters
• Monitor Kubernetes clusters
• Debug Kubernetes clusters
• Network and secure Kubernetes clusters
• Auto-scale and cost-optimize the cluster according to load

This is a great starting point, but it doesn’t actually help developers in any way. Developers only care about shipping features. Kubernetes is a technical detail for them, as virtual machines were before Kubernetes.

The problem is that almost all the tools available focus on individual clusters. If your organization is using any kind of Kubernetes dashboard, I would bet that on the left sidebar there is a nice big button called “clusters” that shows me a list of all available Kubernetes installations.

But here is the hard truth. Developers don’t care about Kubernetes clusters. They care about environments—more specifically the classic trilogy of QA, staging, and production. That’s it.

Maybe in your organization Staging is a single cluster. Maybe Staging is two clusters. Maybe Staging is a namespace inside another bigger cluster. Maybe Staging is even a virtual cluster. It doesn’t really matter for developers. All they want to see is an easy way to deploy their features from one environment to the next.

If you want to make life easy for developers, then offer them what they actually need.

• A list of predefined environments with a logical progression structure
• A way to “deploy” their application to those environments without any deep Kubernetes knowledge
• An easy way to create temporary preview environments for testing a new feature in isolation
• A powerful tool to debug deployments when things go wrong.

In this manner, developers will be able to focus on what actually matters to them. If you force developers to learn Helm, Kustomize, or how Kubernetes manifests work, you are wasting their time. If every time a deployment fails, your answer is “just use kubectl to debug the cluster,” then you are doing it wrong.

Promotions are more critical than deployments

So, let’s say you followed my advice and offered your developers a nice dashboard that presents environments instead of individual clusters. Is that enough?

It turns out that you must also offer a way to “deploy” to those environments. But here is the critical point. Make sure that your fancy dashboard understands the difference between a deployment and a promotion.

A deployment to an environment should be a straightforward process. A developer must be able to choose

• A version of their application (the latest one or a previous one)
• An environment with appropriate access
• A way to make sure that the deployment has finished successfully.

Sounds simple, right? It is simple, but this process is only useful for the *first *environment where code needs to be verified. For the rest of the environments in the chains, developers want to promote their application.

Unlike a deployment, a promotion is a bit more complex. It means that a developer wants to take what is already available in the previous environments (e.g., QA) and move that very same package in the next environment (e.g., Staging).

The magic here is that if you look at all the environments of the organization there is a constant battle between how “similar” two environments are. In the most classic example, your Staging environment should be as close as possible to Production in order to make sure that you test your application in similar conditions.

On the other hand, it should be obvious that Staging should not be able to access your production database or your production queues. You should have separate infrastructure for handling Staging data.

This means that by definition some configuration settings (e.g., database credentials) are different between Production and Staging. So when a developer wants to “promote” an application, what they really want to do is

• Take the parts of the application that actually need to move from one environment to another
• Ignore the configuration settings that stay constant between environments.

This is a very important distinction, and the majority of cloud tools do not understand it. Many production incidents start because either a configuration change was different between production and staging (or whichever was the previous environment) or because the developer deployed the wrong version to production, bypassing the previous environments.

Coming back to your developer dashboard, if you offer your developers just a drop down list from all possible versions of an application allowing them to choose what to deploy, you are doing it wrong. What developers really want is to promote whatever is active and verified in the previous environment.

At least for production this should be enforced at all times. Production deployments are the last step in a chain where a software version is gradually moved from one environment to another.

Behind the scenes, your fancy dashboard should also understand what configuration needs to be promoted and what configuration stays the same for each environment. In the case of Kubernetes, for example, the number of replicas for each environment is probably static. But your application’s configmaps should move from one environment to another when a promotion happens.

Deployment pipelines no longer work in the cloud era

We have covered environments and promotions so it is time to talk how exactly a deployment takes place. The traditional way of deploying an application is via pipelines. Most continuous integration software has a way of creating pipelines as a series of steps (or scripts) that execute one after the other.

The typical pipeline consists of:

1. Checking out the source code of the application
2. Compiling and building the code
3. Running unit and integration tests
4. Scanning the code for vulnerabilities
5. Packaging the code in its final deliverable.

Before the cloud, it was common to have another step in the pipeline that took the binary artifact and deployed it to a machine (via FTP, rsync, SSH, etc.). The problem with this approach is that the pipeline only knows what is happening while** **the pipeline is running. Once the pipeline has finished, it no longer has visibility into what is happening in the cluster.

This creates a very unfortunate situation for developers, with the following pattern:

1. A developer is ready to perform a deployment
2. They start the respective pipeline in the continuous integration dashboard
3. The pipeline runs successfully and deploys the application to the cluster
4. The pipeline ends with a “green” status
5. Five minutes later the application faces an issue (e.g., slow requests, missing database, evicted pods)
6. The developer still sees as “green” the pipeline and has no way of understanding what went wrong.

It is at this point that developers are forced to look at complex metrics or other external systems in order to understand what went wrong. But developers shouldn’t have to look in multiple places to understand if their deployment is OK or not.

Your deployment system should also monitor applications, even after the initial deployment has finished. This is an absolute requirement for cloud environments where resources come and go—especially in the case of Kubernetes clusters, where autoscaling is in constant effect.

Catching up with cloud deployments

Cloud computing comes with its own challenges. Most existing tools were create before the cloud revolution and were never designed for the dynamic nature of how cloud deployments work. In this new era, developers are left behind because nobody really understands what they need.

In the case of Kubernetes, existing tools tend to be oriented towards operators and administrators:

• They show too much low-level information that is not useful for developers
• They don’t understand how environments are different and how to promote applications
• They still have the old mindset of continuous integration pipelines.

We need to rethink how cloud computing affects developers. With the recent surge in generative AI and LLM tools, deploying applications will quickly become the bottleneck. Developers will be able to quickly create features with their smart IDEs or AI agents, but they will never understand how to promote applications or how to quickly pinpoint what was the issue of a failed deployment.

—

New Tech Forum*** provides a venue for technology leaders—including vendors and other outside contributors—to explore and discuss emerging enterprise technology in unprecedented depth and breadth. The selection is subjective, based on our pick of the technologies we believe to be important and of greatest interest to InfoWorld readers. InfoWorld does not accept marketing collateral for publication and reserves the right to edit all contributed content. Send all ***inquiries to doug_dineley@foundryco.com.

by Kostis Kapelonis

Contributor

Kostis Kapelonis is a software engineer with a special focus on GitOps and deployment automation. He currently serves as senior developer advocate for Octopus Deploy, maintainer of the Argo open source project, and co-author of the world’s first Argo CD certification. He is also co-author of a Manning book, Java Testing with Spock.