I would like to keep this moving forward. What do you suggest as next steps?
I’m thinking a few separate tasks:
Post on the Open edX community forums + Slack to ask (A) who is using non-AWS/Google/Azure hosting, and how their experience has been, and (B) who is interested in collaborating on support for deploying large/many Open edX instances on other clouds or infra.
Make a short-list of alternative providers, ideally limiting ourselves for now to those who offer Managed K8S + Managed MySQL/MariaDB + object storage + Terraform. That criteria should shorten the list quite a bit.
It could be worth investigating options for deploying databases when managed databases aren’t available from the provider, or simply to make our database deployments consistent across all non-AWS providers. (For example, if we were using PostgreSQL, Pigsty [“open source RDS”] looks like an ideal way to manage it on any VM or even bare metal hosting service. For MySQL/MongoDB I don’t know if there’s anything as nice, but Percona Operator for MongoDB looks pretty good for MongoDB. Same for managed storage: Garage or RustFS. This could be deferred until we have a client [Ethiopia?] that requires it, but it could be good to do a timeboxed investigation now so we have some options in hand.) It doesn’t have to be all or nothing either - we could decide that we always deploy our own MongoDB on clusters using the Percona Operator, but rely on managed services for Object Storage + MySQL. (Since those are commonly available, but MongoDB is not.)
Something that may be worth looking at here is Juju. It brings its own complexity, but does have operators for many databases, both on K8s and VMs/bare metal.
Anyone wants to take this? If nobody volunteers until Wednesday, I can do it.
I support this! Rolling our own things would require way more effort and probably not something that most of the community members would want to do anyway.
@antoviaque I have scheduled it for this sprint to sort this out. In the past two sprints, I was working on making our infrastructure more portable, decoupled from terraformed and external services on several levels as much as possible, so whatever the decision is in the future, we have way less work. Also, the maintenance time is going to drop while reliability is going up.
@gabor Thank you - do you have the ticket URL to follow?
Also, relatedly – how about the community discussion, and the OEP? It would be important to follow-up actively on those - and have a draft of the OEP for community review? That would have priority currently over further internal improvements to our infrastructure - we need to collaborate more ahead of the dev & devops work itself going forward.
If you mean to reschedule the meeting to Wednesday, it was completed by Axim Engineering yesterday. From the next meeting (2 weeks from now) it will be on Wednesdays.
and the OEP
We clarified with eduNEXT which OEP they meant and as part of the linked ticket, the update proposal will be submitted. Also, we discuseed with them that they are absolutely in favor of extending that OEP and they appreciate the effort on that.
It would be important to follow-up actively on those
Yes, I agree and we are pushing that forward.
That would have priority currently over further internal improvements to our infrastructure
I agree here as well, though there were some “timing”/”prerequisites”. All the new things we introduced and “experimented” with on internal infra will be reflected on the Tutor, Harmony and/or Launchpad side. Which will include the following:
“Provider-agnostic” DNS record setup and automatic maintenance using external-dns
Secret management using external-secrets which means the cluster repositories and instance configs will not (or minimally?) contain secrets, so most probably can be open-sourced as needed while not giving up on security.
Nginx → Traefik replacement (in progress) which is required as Nginx is now a security risk as not being updated for Kubernetes
Setting up remotely managed ArgoCD instances (i.e., we can oversee all cluster’s instance’s health from one place while the clusters are completely independent and can be deployed as they needed – imagine OCIM without ansible, running on steroids with no extra effort)
Independant, but centrally observed monitoring and alerting for the clusters, so we oversee everything for clusters with alerting in place, but not not loosing insights if the central Grafana goes down.
Discovery is updated as well with Cleura, Akamai Linode, Scaleway, and UpCloud. We can reduce the costs of MySQL database if we really want to self-manage but that price is not included in the estimates. Approximately on a ~$80-120/month cost.
Thanks for updating the discovery. The cost for almost all of them seems to be within ~10% of each other, which is to be expected for such a commoditized product as K8S hosting.
So I think cost is less of a concern, and we need to focus on the other factors like reliability, support, capacity, etc.
@braden I have no preference. If I would have to choose for myself, I would go with:
AWS if hyperscaler: they have saving plans and the Open edX community is heavily depending on it.
UpScale if non-hyperscaler: their product seemed to be the most polished so far, without playing around with their stack.
If we count Linode as non-hyperscaler, I would choose between UpScale and Linode by tossing a coin.
However, if we select one, I would say to setup a test cluster with (any of) them (if not AWS) and put some load on it for testing. Also, test their support team. If the cluster is at least that performant as DigitalOcean, their support is better, and their developer experience is good, I think we found our “new home”. Otherwise, it turned out in a non-prod environment that they are not reliable.
@gabor@braden So we do this? ^ We will still need AWS for some clients who require it specifically, but for the rest of the infra & instances we aim to switch to either UpCloud or Linode, after load-testing them and testing their support? Pick either to start, and if it doesn’t work out we try the other one?
@antoviaque I think we can close the discovery. Although I’m waiting for a review from @braden & @mtyaka, I was able to setup a Kubernetes cluster with Launchpad and Harmony and deploy an instance on UpCloud. The infra setup was way smoother and better experience than DigitalOcean; also I “tested” their support response time which was excellent plus the servers. From the very basic load tests, it seems that the final setup will be somewhat more performant as well.
If you agree with me, once SE-6721 is done, I’m going to close the discovery (SE-6606) as well and open a follow-up Epic detailing the move from DigitalOcean to UpCloud.
I’ll sort it out this week.