I would like to keep this moving forward. What do you suggest as next steps?
I’m thinking a few separate tasks:
Post on the Open edX community forums + Slack to ask (A) who is using non-AWS/Google/Azure hosting, and how their experience has been, and (B) who is interested in collaborating on support for deploying large/many Open edX instances on other clouds or infra.
Make a short-list of alternative providers, ideally limiting ourselves for now to those who offer Managed K8S + Managed MySQL/MariaDB + object storage + Terraform. That criteria should shorten the list quite a bit.
It could be worth investigating options for deploying databases when managed databases aren’t available from the provider, or simply to make our database deployments consistent across all non-AWS providers. (For example, if we were using PostgreSQL, Pigsty [“open source RDS”] looks like an ideal way to manage it on any VM or even bare metal hosting service. For MySQL/MongoDB I don’t know if there’s anything as nice, but Percona Operator for MongoDB looks pretty good for MongoDB. Same for managed storage: Garage or RustFS. This could be deferred until we have a client [Ethiopia?] that requires it, but it could be good to do a timeboxed investigation now so we have some options in hand.) It doesn’t have to be all or nothing either - we could decide that we always deploy our own MongoDB on clusters using the Percona Operator, but rely on managed services for Object Storage + MySQL. (Since those are commonly available, but MongoDB is not.)
Something that may be worth looking at here is Juju. It brings its own complexity, but does have operators for many databases, both on K8s and VMs/bare metal.
Anyone wants to take this? If nobody volunteers until Wednesday, I can do it.
I support this! Rolling our own things would require way more effort and probably not something that most of the community members would want to do anyway.
@antoviaque I have scheduled it for this sprint to sort this out. In the past two sprints, I was working on making our infrastructure more portable, decoupled from terraformed and external services on several levels as much as possible, so whatever the decision is in the future, we have way less work. Also, the maintenance time is going to drop while reliability is going up.
@gabor Thank you - do you have the ticket URL to follow?
Also, relatedly – how about the community discussion, and the OEP? It would be important to follow-up actively on those - and have a draft of the OEP for community review? That would have priority currently over further internal improvements to our infrastructure - we need to collaborate more ahead of the dev & devops work itself going forward.
If you mean to reschedule the meeting to Wednesday, it was completed by Axim Engineering yesterday. From the next meeting (2 weeks from now) it will be on Wednesdays.
and the OEP
We clarified with eduNEXT which OEP they meant and as part of the linked ticket, the update proposal will be submitted. Also, we discuseed with them that they are absolutely in favor of extending that OEP and they appreciate the effort on that.
It would be important to follow-up actively on those
Yes, I agree and we are pushing that forward.
That would have priority currently over further internal improvements to our infrastructure
I agree here as well, though there were some “timing”/”prerequisites”. All the new things we introduced and “experimented” with on internal infra will be reflected on the Tutor, Harmony and/or Launchpad side. Which will include the following:
“Provider-agnostic” DNS record setup and automatic maintenance using external-dns
Secret management using external-secrets which means the cluster repositories and instance configs will not (or minimally?) contain secrets, so most probably can be open-sourced as needed while not giving up on security.
Nginx → Traefik replacement (in progress) which is required as Nginx is now a security risk as not being updated for Kubernetes
Setting up remotely managed ArgoCD instances (i.e., we can oversee all cluster’s instance’s health from one place while the clusters are completely independent and can be deployed as they needed – imagine OCIM without ansible, running on steroids with no extra effort)
Independant, but centrally observed monitoring and alerting for the clusters, so we oversee everything for clusters with alerting in place, but not not loosing insights if the central Grafana goes down.
I’ll sort it out this week.