As my reply is quite long this time, I split it into four sections based on the scope, so it is easier to oversee it.
1. Fragmented knowledge
Lately, by reading the forum posts and Jira comments, I started to build the feeling that we are having issues with finding the written knowledge we have. Even though a lot of human-ops and process related topics are described in the handbook, many technical documentation exist in the relevant repositories.
The conversation and documentation is fragmented between: Mattermost (it shouldn’t be the case on paper), forum, Jira, technical docs, handbook, private docs, GitHub and GitLab READMEs, OpenCraft Course, Grove documentation, Launchpad documentation, Google Docs-based discoveries, and “random” repository documentation whether it is upstream or owned by OpenCraft (e.g., ansible-playbooks repo, infrastructure repo, etc.).
As this is not the first and probably not the last instance of a longer conversation that has reasoning split across multiple places, I propose to start a discovery to handle this issue if this is technical. As our documentation base grows, it is proven that it is less likely to find the information we are looking for or simply the effort and time was not put in.
2. Launchpad stack development history and communication
On another note, we’ve had multiple discussions about the Launchpad stack, both internally and with members of the Large Instances Working Group (EduNext). The intended audience (operators of instances on scale) was already engaged with the topic before these discoveries were made. In addition to those conversations, both within the working group and in standalone meetings, we conducted an initial discovery to identify the issues with Grove, followed by a more detailed discovery that outlined the desired solution while taking into account the current constraints we are working within.
Following the discovery, we had ongoing forum discussions covering implementation details (ref1, ref2, and more down below in the thread), shared videos demonstrating the earliest stages of development, and provided regular updates (ref1, ref2) through the epic and forum posts (ref1, ref2, ref3, ref4) as work progressed. After discoveries were accepted, we also conducted additional discoveries (ref1, ref2) work based on the new stack, which now serves as the foundation for Axim’s sandbox implementation.
One of the primary reasons for setting up Launchpad on GitHub was to move away from GitLab CI that causes us a LOT of reliability issues, enable better collaboration with the community, lower the barrier to contribution, and allow the reuse of existing code such as Picasso and Drydock, which are already developed and maintained on GitHub (by EduNext).
Note: I can highlight only EduNext as for a very long time, they are the only people joining the Large Instances Working Group meetings. This seems to change slowly as we have one or two new people for the last 2 meetings (meaning from mid-July). Before that, noone else were interested in discussions about instance operations at scale.
3. Moving to GitLab for Launchpad and cluster repos
I already discussed this with Samuel, but I’ll try to conclude that part of the conversation here as well.
The stack is building on GitHub actions in a way that Picasso and Drydock provides the images utilizing the Tutor commands. Based on our experiences with Grove in regards to GitLab CI reliability and other issues, moving back to GitLab CI would be a vast step back. Even if the idea of closed source software is not liked, we spent a lot of time and effort on fixing GitLab CI related user issues. Anyone in the team who had to deal with GitLab CI issues are praising the new setup as I see so far.
Regarding how frequently some functions are running on GitHub, let me make a quick correction: The instance create/update/delete pipelines are not rarely used. Those are used multiple times every day by the sandboxes integration (mostly used by Axim, but by OpenCraft as well). It is a crucial part of the sandbox implementations. The number of sandbox provisioning, reliability, and CI-related issues dramatically reduced since the new stack. This is due to the effort @Kaustav and I put into the Axim sandbox provisioning and decoupling, the CI improvements (that are technically not possible on GitLab CI), and the new stack’s primitives (e.g., Picasso, Drydock, ArgoCD, GitHub Actions, and more).
Also, Drydock is building Kubernetes manifests for us and configures ArgoCD syncwaves that determines the order of resource synchronization on the Kubernetes clusters during deployment. This sounds a tiny bit, but in reality, to match everything on GitLab would mean 100+ hours spent on duplicating an already existing logic and maintain two different versions of Picasso and Drydock. This is something we deliberatly avoided and this (read: reducing duplications and unify tooling as much as possible) was the starting discussion of the discoveries.
4. Moving and creating repos to GitLab
As the handbook says, we should create new repositories on GitLab unless there is a strong reason not to. Moving repositories that can be moved to GitLab already happened once as I remember, but if other subjects found to get moved, let’s move them (with the exception of Launchpad or sandbox automation related repos).
