Hi, I have been working with the Launchpad stack recently and I noticed something - each cluster serves only one environment. As in, either staging or production.
I am really puzzled by this design. It’s a kubernetes cluster. Aren’t they supposed to be resilient and be able to host multiple environments in separate namespaces without affecting one another?
I didn’t quite understand when we had grove-staging and grove-production as well, but that got eventually consolidated to just grove-production.
In our shared cluster instances are starting to appear like “my-instance-stage-production”, “my-instance-prod-production” because the cluster is marked as “production” environment. While it is only cosmetically strange for the shared cluster, I see some clients having dedicated Github repos for the 2 environments with their “instances” directory with just “staging” instance or “production” instance. It feels a bit like a glorified docker-compose at that point.
Is this the result of our inability to stop adding abstraction layers? Or a symptom of the cloud providers’ reliability going down? What’s driving this design where we are treating entire k8s clusters as an individual unit of compute?
I am sorry, I don’t have a ticket for this. Just wanted to ask after spending sometime with the stack to understand things.
@tecoholic the “environment” was something we carried from the Grove setup for Launchpad clusters as well. Which may be quite confusing as I see.
We don’t have abstraction issues as all instance can be configured the way we want. We can have a different database cluster for each instance if we want or even use in-namespace databases for some instances while we use managed database for the rest. This is the same with everything as the instances are not depending on how the cluster is set up (read: besides the fact that the instance is running on the cluster of course).
The “environment” word is confusing as you wrote as we could deploy production, staging, dev, PR, etc. instances for a single cluster if we want. What I would say is that we could either do:
Treat all clusters that cannot be recreated anytime as production – clusters that we can tinker with could be dev, staging, or anything else. OR
Let’s rename the “environment” tag.
This is just a label we use to identify the purpose of clusters easier, not necessarily the purpose of the instances running on them.
@gabor Thanks for explaining the situation. Yeah, renaming or even just decoupling the cluster environment from the instance’s environment will be helpful in our shared cluster.
I am still wondering about this:
I think this statement kind of covers it -
If I understand correctly, this is the effect of the “cluster” itself being a development target due to IaaC tools like Terraform, right?
I’m not entirely sure about every cluster as I was involved only in the shared clusters. However, I doubt that those clusters should be kept only for staging instances. We are rarely changing infrastructure under normal circumstances other than upgrading the cluster or services running along (e.g., cert-manager).
If a client wants to reduce the possibility of hiccups and not sensitive to ~1.5x-2x the monthly costs, sure we can keep 2 clusters (staging/dev and prod). But truthfully, I would keep around one cluster and deploy the 1, 2, 3, N+ environments (dev, stage, prod, anything else) there. Each instance is resource-limited so they cannot crash a cluster, each instance has its own namespace, the instances can be configured separately for database and storage access (and for everything else).
Now, Launchpad supports extra terraform scripts added in the respective infrastructure directory (e.g., here) and we have modules in Harmony and OpenCraft to setup extra databases if needed to separate DB clusters based on need.
So, that’s not an “excuse” anymore to have one cluster, one cluster repo, and multiple databases or buckets as needed.
This is good to know. Maybe we need to have a wider discussion about this, as I think there are different understandings among the team. I was advised that it’s still worth the extra cost to get the extra stability with separate k8s clusters for stage and prod.
It would be good to have a clear cost and risk breakdown of the two options, so clients can make an informed decision.
In a single-cluster environment, Harmony is a single point of failure. It is deployed into a single namespace that manages all instances in the cluster. As a result, larger changes, like the recent switch from NGINX Ingress to Traefik, cannot be tested independently in a separate environment. Moreover, Harmony still doesn’t have a stable version (it’s currently at 0.16.0) and has no changelog, so upgrading requires reviewing the commits. Especially since minor changes, like this one, require manual changes on the cluster. The fact that the changes are concealed as chore(deps) and the required steps are mentioned only in the PR description doesn’t give confidence in the stability of these modules.
Our terraform-modules could also use a changelog to understand what’s happening and which Harmony version it is compatible with, without checking it directly in the deps.
While I would like to reduce maintenance overhead and put all envs in a single cluster/repo, it still feels like the cost of maintaining an additional cluster to test rolling out updates is lower than the risk of bringing down the prod cluster. Having an established (and stable) semver with integration tests for the critical Terraform modules would be great, but we may be a bit far from that for now.