This is a thread to serve as a central location to report on the latest updates to build status for Open edX instances and devstack. For example, if current master devstack is broken, current required workarounds, etc.
Periodic Build Master is an instance configured for periodic builds and running of upstream master for edx-platform and configuration. If this is green, then new master instances are likely to be fine. If not, check here to see if there are known issues and/or workarounds.
We are also trialing a new master breakage reporting to edX process - see SE-1993.
[ Time spent on this type of issue and this forum thread should be logged on the specific ticket that generated the PR/sandbox, or for larger issues, on tickets specific to that problem. ]
Currently, master docker devstacks are building fine.
Master instances on OCIM are currently broken because of breaking changes introduced to configuration in edx/configuration#5414. This is being worked on and discussed in BB-2135.
For now a workaround involves a configuration commit before #5414 and extra configuration: See this post below.
To continue the tradition of mentioning the ticket where to log time in forum posts: … any ticket where to log time when discussing generic upstream breakages?
@daniel It doesn’t make sense to have a catch-all ticket for these issues. Time should be logged on the specific ticket that generated the PR/sandbox, or for larger issues, on tickets specific to that problem.
Redeployed the LabXchange CI instance yesterday to quiet alerts for SE-2161, and it’s working with a near-master pinned version of the configuration repo, plus a bunch of workaround settings.
# Braden’s temporary fix 2020-01-29 to work around https://github.com/edx/configuration/pull/5414#issuecomment-579997504 # -> configuration version pinned to 5806881b0071aeef355054e7b658855fc22e4035 instead of ‘master’ # Braden’s temporary fix for configuration bug - remove when edX fixes it or removes the prospectus code https://github.com/edx/configuration/pull/5561 PROSPECTUS_PREVIEW_NGINX_USERS: [] PROSPECTUS_PREVIEW_HTPASSWD_PASS: "doesntmatter!" PROSPECTUS_PREVIEW_HTPASSWD_USER: “dontuse” # Braden’s other temporary fix 2020-01-23 # https://chat.opencraft.com/opencraft/pl/4ckt5atqkbyzbmddcx8q93bw8h EDXAPP_VIDEO_IMAGE_SETTINGS: VIDEO_IMAGE_MAX_BYTES : 2097152 VIDEO_IMAGE_MIN_BYTES : 2048 STORAGE_KWARGS: location: “video-images/” bucket: “{{ EDXAPP_AWS_STORAGE_BUCKET_NAME }}” # https://chat.opencraft.com/opencraft/pl/4ckt5atqkbyzbmddcx8q93bw8h EDXAPP_VIDEO_TRANSCRIPTS_SETTINGS: VIDEO_TRANSCRIPTS_MAX_BYTES : 3145728 STORAGE_KWARGS: location: “video-transcripts/” bucket: “{{ EDXAPP_AWS_STORAGE_BUCKET_NAME }}” # Jill’s fix from 2020-03-02 # adds change from https://github.com/edx/configuration/pull/5649 forum_ruby_version: “2.5.7” # Other fixes ICP_LICENSE_INFO: {} AWS_GATHER_FACTS: false
The last two are already in our default configuration settings for the OpenCraft edx:edx-platform watched fork, which means they’re automatically set on all new PR sandboxes.
Does anyone have any objections to me updating this watched fork to use the above settings and pinning the configuration version, so our new PR sandboxes will at least build?
If anyone has issues with a new PR sandbox, or with an existing one updated with these settings, could you post here and we’ll work out what else needs to be fixed?
to get around this error on his sandbox, which is running with a slightly different configuration than we’ve discussed here:
django.core.exceptions.ImproperlyConfigured: S3BotoStorage.location cannot begin with a leading slash. Found ‘/edx/var/edxapp/media/’. Use ‘edx/var/edxapp/media/’ instead.
Unknown command: 'create_oauth2_client
Type ‘manage.edxapp help’ for usage.
Upstream have removed the Django OAuth2 Provider libraries with edx-platform, but since we’re using an old fork of the configuration repo to try to stay afloat, we didn’t get the corresponding change in the configuration repo, so we’re still trying to call the now-nonexistant-in-master create_oauth2_client management command.
I’ve created SE-2307 to address this next sprint. Any takers, or better ideas welcome!
It’s based on the configuration version 5806881 from my previous comment, and cherry-picks in the changes from configuration#5635. It’s currently working for building master sandboxes (e.g. this one), so I’ve updated the new PRs watched fork to use it, and will update the other affected failing sandboxes too.
But given the major upheavals with master recently, this workaround may not work for long. Could the next person who has a PR sandbox fail please post here with the error, and either fix it by pushing the fix to that branch, or create a ticket to do so if it’s not straightforward?
This should be temporary, but we’re waiting on edX DevOps to share their plans for the configuration repo, and for us to have time to update Ocim to deal with them cf above.
Found yet another issue, which is causing Studio logins to fail on PR sandboxes: edx-platform#22416 removed the deprecated separate Studio login feature (DISABLE_STUDIO_SSO_OVER_LMS), and so attempts to Signin or Signup from Studio redirect to LMS signin, then back to the Studio index page without authenticating to Studio.
To fix this, we need to allow the LMS and Studio to share session cookies, by setting:
I’ve updated the PR watched fork extra configuration to set this, but have not updated the existing PR sandboxes. Anyone experiencing this issue should add the above config and rebuild their sandbox.
Master sandboxes are building now, but there’s authentication issues with the forums, which causes these error dialogs to appear when there are inline forum units:
And visiting the Discussion link shows our error screen.
Luckily, edx-platform master still works with the ironwood.2 forum code, so have updated our watched fork to use this instead:
# Jill’s fix from 2020-03-02
# adds change from https://github.com/edx/configuration/pull/5649
# forum_ruby_version: "2.5.7"
# Update 2020-03-22 issues with cs_comments_service mean we can't use latest master,
# so removed above change, and added this:
forum_version: open-release/ironwood.2
@jill, @kshitij, we (@giovannicimolin and I) had fixed the forum MongoDB authentication issues by setting the FORUM_MONGO_AUTH_DB to the name of the forum database itself (used as the value of FORUM_MONGO_DATABASE) when we ran into similar errors on the Rue89 instance with mLab. I am not sure if the issue is the same as I haven’t checked the logs yet.
Oo that’s interesting… I’m going to leave the forum version at ironwood.2 for new PR sandboxes until we have further issues with the forum, since there is still a lot of back-and-forth merging and reverting happening on the master branch there. But will try that next, thank you!
The https://github.com/edx/configuration/pull/5691 PR is renaming the *_version ansible variables to upper case. Once it is merged, this will likely break our master sandboxes and will require Ocim changes for Juniper (CC @toxinu).
The EDXAPP_VIDEO_IMAGE_SETTINGS and EDXAPP_VIDEO_TRANSCRIPTS_SETTINGS either need an upstream PR, or an Ocim change to apply them to all our instances.
EDITThe forum_version has been bumped to open-release/juniper.alpha1, since there’s stillWe’ve merged PRs to resolve the issues with our mongo and master with upstream PRs, and will be addressing the authentication mechanism update in the follow-up tasks for SE-2383. I’ve updated the watched fork and notes below accordingly.
s and no objections, so I’ve 