Our Openedx Instance Test Checklist has grown in size significantly since I last used it, and it takes quite a long time to go through it now. Does it make sense to try automating it with tools like Cypress or Playwright?
Unlike Selenium-based tests, both tools support recording browser sessions, so they can be reviewed conveniently, excluding possible undetected issues.
Client owners that deploy frequently: @kaustav (@paulo), @Agrendalath — would that kind of automation be helpful to you, provided you can easily extend it to test client-specific features?
That is a very interesting idea, and it occurred to me in the past as well.
Having good end to end test can help improve the stability and reduce regression after upgrades. We could potentially even write tests for custom features of each client. I would’ve loved it for some of our clients in the past.
That being said, the amount of effort it takes to implement and maintain these tests is very easy to underestimate. Because the end to end tests have to interact with the UI and depend on network, the tests can be very flaky. Also, you will have to make test suits be flexible for all kinds of clients:
If forums are disabled, the tests for the forum shouldn’t run. Same if they are using a different forum provider.
If the client has TPA or a different end point for logging in, the tests have to be configurable to be able to account for that.
I honestly think this would be a great idea, but we needn’t make it an all or nothing thing. We can start by automating some very simple things with a very stable interface, and take it from there.
In any case on a project like this, it would be worth discussing it with the larger community, to make sure it’s useful to the project as a whole, and is developed as an official approach (ie included in the upstream CI/CD pipeline) - not just something internal to us. I agree that it is indeed easy to underestimate the maintenance work on these.
I like the idea of implementing smoke (or even regression) tests. We discussed a similar thing with one of our clients a few years ago but never prioritized this, as the instance has been too stable to justify this. It would be awesome if we could find a way to develop this incrementally and involve the community.