Open edX devstack and Ocim sandbox issues

swalladge · March 2, 2020, 1:17am

This is a thread to serve as a central location to report on the latest updates to build status for Open edX instances and devstack. For example, if current master devstack is broken, current required workarounds, etc.

Periodic Build Master is an instance configured for periodic builds and running of upstream master for edx-platform and configuration. If this is green, then new master instances are likely to be fine. If not, check here to see if there are known issues and/or workarounds.

We are also trialing a new master breakage reporting to edX process - see SE-1993.

[ Time spent on this type of issue and this forum thread should be logged on the specific ticket that generated the PR/sandbox, or for larger issues, on tickets specific to that problem. ]

swalladge · March 2, 2020, 1:22am

Currently, master docker devstacks are building fine.

Master instances on OCIM are currently broken because of breaking changes introduced to configuration in edx/configuration#5414. This is being worked on and discussed in BB-2135.

For now a workaround involves a configuration commit before #5414 and extra configuration: See this post below.

daniel · March 2, 2020, 1:39am

To continue the tradition of mentioning the ticket where to log time in forum posts: … any ticket where to log time when discussing generic upstream breakages?

swalladge · March 2, 2020, 3:22am

To be honest, I’m not 100% clear on whether this will be part of the periodic builds epic or not. Maybe @tikr or @antoviaque have some ideas.

jill · March 2, 2020, 3:40am

@daniel It doesn’t make sense to have a catch-all ticket for these issues. Time should be logged on the specific ticket that generated the PR/sandbox, or for larger issues, on tickets specific to that problem.

antoviaque · March 2, 2020, 5:51am

@swalladge Thank you for creating this thread!

+1 - I’ll add this to the bottom of the first post on this thread, for clarity.

jill · March 3, 2020, 1:30am

Redeployed the LabXchange CI instance yesterday to quiet alerts for SE-2161, and it’s working with a near-master pinned version of the configuration repo, plus a bunch of workaround settings.

name	value
configuration_source_repo_url	`https://github.com/edx/configuration`
configuration_version	`5806881b0071aeef355054e7b658855fc22e4035`
configuration_playbook_name	playbooks/openedx_native.yml
edx_platform_repository_url	`https://github.com/edx/edx-platform`
edx_platform_commit	`release`
openedx_release	`master`
configuration_extra_settings	# Braden’s temporary fix 2020-01-29 to work around https://github.com/edx/configuration/pull/5414#issuecomment-579997504 # -> configuration version pinned to 5806881b0071aeef355054e7b658855fc22e4035 instead of ‘master’ # Braden’s temporary fix for configuration bug - remove when edX fixes it or removes the prospectus code https://github.com/edx/configuration/pull/5561 PROSPECTUS_PREVIEW_NGINX_USERS: [] PROSPECTUS_PREVIEW_HTPASSWD_PASS: "doesntmatter!" PROSPECTUS_PREVIEW_HTPASSWD_USER: “dontuse” # Braden’s other temporary fix 2020-01-23 # https://chat.opencraft.com/opencraft/pl/4ckt5atqkbyzbmddcx8q93bw8h EDXAPP_VIDEO_IMAGE_SETTINGS: VIDEO_IMAGE_MAX_BYTES : 2097152 VIDEO_IMAGE_MIN_BYTES : 2048 STORAGE_KWARGS: location: “video-images/” bucket: “{{ EDXAPP_AWS_STORAGE_BUCKET_NAME }}” # https://chat.opencraft.com/opencraft/pl/4ckt5atqkbyzbmddcx8q93bw8h EDXAPP_VIDEO_TRANSCRIPTS_SETTINGS: VIDEO_TRANSCRIPTS_MAX_BYTES : 3145728 STORAGE_KWARGS: location: “video-transcripts/” bucket: “{{ EDXAPP_AWS_STORAGE_BUCKET_NAME }}” # Jill’s fix from 2020-03-02 # adds change from https://github.com/edx/configuration/pull/5649 forum_ruby_version: “2.5.7” # Other fixes ICP_LICENSE_INFO: {} AWS_GATHER_FACTS: false

The last two are already in our default configuration settings for the OpenCraft edx:edx-platform watched fork, which means they’re automatically set on all new PR sandboxes.

Does anyone have any objections to me updating this watched fork to use the above settings and pinning the configuration version, so our new PR sandboxes will at least build?

jill · March 4, 2020, 5:28am

Okee, I got two s and no objections, so I’ve updated our watched fork.

If anyone has issues with a new PR sandbox, or with an existing one updated with these settings, could you post here and we’ll work out what else needs to be fixed?

jill · March 5, 2020, 1:03am

FYI @paulo reports on BB-527 that he had to add

edxapp_media_dir: media

to get around this error on his sandbox, which is running with a slightly different configuration than we’ve discussed here:

django.core.exceptions.ImproperlyConfigured: S3BotoStorage.location cannot begin with a leading slash. Found ‘/edx/var/edxapp/media/’. Use ‘edx/var/edxapp/media/’ instead.

jill · March 6, 2020, 3:23am

Ok, beware when copy pasting the above config – Discourse markdown puts smart quotes into code blocks. Copy from our watched fork if you need to.

jill · March 6, 2020, 2:56pm

Another major breaking change, which has rendered the above pinned configuration branch obsolete, as @kahlil discovered on SE-2218

Unknown command: 'create_oauth2_client
Type ‘manage.edxapp help’ for usage.

Upstream have removed the Django OAuth2 Provider libraries with edx-platform, but since we’re using an old fork of the configuration repo to try to stay afloat, we didn’t get the corresponding change in the configuration repo, so we’re still trying to call the now-nonexistant-in-master create_oauth2_client management command.

I’ve created SE-2307 to address this next sprint. Any takers, or better ideas welcome!

jill · March 13, 2020, 8:41am

As per SE-2307, I’ve created a new branch of the configuration repo which addresses this: open-craft:ocim-master

It’s based on the configuration version 5806881 from my previous comment, and cherry-picks in the changes from configuration#5635. It’s currently working for building master sandboxes (e.g. this one), so I’ve updated the new PRs watched fork to use it, and will update the other affected failing sandboxes too.

But given the major upheavals with master recently, this workaround may not work for long. Could the next person who has a PR sandbox fail please post here with the error, and either fix it by pushing the fix to that branch, or create a ticket to do so if it’s not straightforward?

This should be temporary, but we’re waiting on edX DevOps to share their plans for the configuration repo, and for us to have time to update Ocim to deal with them cf above.

jill · March 15, 2020, 5:42am

Found yet another issue, which is causing Studio logins to fail on PR sandboxes: edx-platform#22416 removed the deprecated separate Studio login feature (DISABLE_STUDIO_SSO_OVER_LMS), and so attempts to Signin or Signup from Studio redirect to LMS signin, then back to the Studio index page without authenticating to Studio.

To fix this, we need to allow the LMS and Studio to share session cookies, by setting:

EDXAPP_SESSION_COOKIE_DOMAIN: ".{{ EDXAPP_LMS_BASE }}"

I’ve updated the PR watched fork extra configuration to set this, but have not updated the existing PR sandboxes. Anyone experiencing this issue should add the above config and rebuild their sandbox.

jill · March 18, 2020, 4:40am

Master sandboxes are building now, but there’s authentication issues with the forums, which causes these error dialogs to appear when there are inline forum units:

forum load error

And visiting the Discussion link shows our error screen.

Click to see forum traceback

==> /edx/var/log/supervisor/forum-stderr.log <==
I, [2020-03-18T04:17:46.741862 #2435]  INFO -- : source=rack-timeout id=517507b5a94017882261f7322061cdec timeout=20000ms state=ready
2020-03-18 04:17:46 - Mongo::Auth::Unauthorized - User pprhsrluhwcjmpjd is not authorized to access pr23359_sandbox_opencraft_hosting_forum.:
	/edx/app/forum/.gem/ruby/2.5.0/gems/mongo-2.5.3/lib/mongo/auth/scram/conversation.rb:489:in `validate!'
	/edx/app/forum/.gem/ruby/2.5.0/gems/mongo-2.5.3/lib/mongo/auth/scram/conversation.rb:477:in `validate_final_message!'
	/edx/app/forum/.gem/ruby/2.5.0/gems/mongo-2.5.3/lib/mongo/auth/scram/conversation.rb:151:in `finalize'
	/edx/app/forum/.gem/ruby/2.5.0/gems/mongo-2.5.3/lib/mongo/auth/scram.rb:63:in `login'
	/edx/app/forum/.gem/ruby/2.5.0/gems/mongo-2.5.3/lib/mongo/server/connection.rb:249:in `block in authenticate!'
	/edx/app/forum/.gem/ruby/2.5.0/gems/mongo-2.5.3/lib/mongo/server.rb:267:in `handle_auth_failure!'
	/edx/app/forum/.gem/ruby/2.5.0/gems/mongo-2.5.3/lib/mongo/server/connection.rb:248:in `authenticate!'
	/edx/app/forum/.gem/ruby/2.5.0/gems/mongo-2.5.3/lib/mongo/server/connection.rb:100:in `connect!'
	/edx/app/forum/.gem/ruby/2.5.0/gems/mongo-2.5.3/lib/mongo/server/connectable.rb:84:in `ensure_connected'
	/edx/app/forum/.gem/ruby/2.5.0/gems/mongo-2.5.3/lib/mongo/server/connection.rb:270:in `write'
	/edx/app/forum/.gem/ruby/2.5.0/gems/mongo-2.5.3/lib/mongo/server/connection.rb:229:in `deliver'
	/edx/app/forum/.gem/ruby/2.5.0/gems/mongo-2.5.3/lib/mongo/server/connection.rb:144:in `block in dispatch'
	/edx/app/forum/.gem/ruby/2.5.0/gems/mongo-2.5.3/lib/mongo/monitoring/publishable.rb:47:in `publish_command'
	/edx/app/forum/.gem/ruby/2.5.0/gems/mongo-2.5.3/lib/mongo/server/connection.rb:143:in `dispatch'
	/edx/app/forum/.gem/ruby/2.5.0/gems/mongo-2.5.3/lib/mongo/operation/shared/executable.rb:33:in `block in dispatch_message'
	/edx/app/forum/.gem/ruby/2.5.0/gems/mongo-2.5.3/lib/mongo/server/connection_pool.rb:110:in `with_connection'
	/edx/app/forum/.gem/ruby/2.5.0/gems/mongo-2.5.3/lib/mongo/server.rb:251:in `with_connection'
	/edx/app/forum/.gem/ruby/2.5.0/gems/mongo-2.5.3/lib/mongo/operation/shared/executable.rb:32:in `dispatch_message'
	/edx/app/forum/.gem/ruby/2.5.0/gems/mongo-2.5.3/lib/mongo/operation/find/command.rb:41:in `execute'
	/edx/app/forum/.gem/ruby/2.5.0/gems/mongo-2.5.3/lib/mongo/operation/find.rb:45:in `execute'
	/edx/app/forum/.gem/ruby/2.5.0/gems/mongo-2.5.3/lib/mongo/collection/view/iterable.rb:82:in `send_initial_query'
	/edx/app/forum/.gem/ruby/2.5.0/gems/mongoid-7.0.5/lib/mongoid/query_cache.rb:232:in `block in each'
	/edx/app/forum/.gem/ruby/2.5.0/gems/mongo-2.5.3/lib/mongo/retryable.rb:44:in `read_with_retry'
	/edx/app/forum/.gem/ruby/2.5.0/gems/mongoid-7.0.5/lib/mongoid/query_cache.rb:230:in `each'
	/edx/app/forum/.gem/ruby/2.5.0/gems/mongoid-7.0.5/lib/mongoid/contextual/mongo.rb:279:in `first'
	/edx/app/forum/.gem/ruby/2.5.0/gems/mongoid-7.0.5/lib/mongoid/contextual/mongo.rb:279:in `find_first'
	/edx/app/forum/.gem/ruby/2.5.0/gems/mongoid-7.0.5/lib/mongoid/contextual.rb:20:in `find_first'
	/edx/app/forum/.gem/ruby/2.5.0/gems/mongoid-7.0.5/lib/mongoid/findable.rb:114:in `find_by'
	/edx/app/forum/cs_comments_service/lib/helpers.rb:13:in `user'
	/edx/app/forum/.gem/ruby/2.5.0/gems/newrelic_rpm-5.6.0.349/lib/new_relic/agent/method_tracer.rb:329:in `block in user_with_trace_Custom_Sinatra_Application_user'
	/edx/app/forum/.gem/ruby/2.5.0/gems/newrelic_rpm-5.6.0.349/lib/new_relic/agent/method_tracer_helpers.rb:30:in `trace_execution_scoped'
	/edx/app/forum/.gem/ruby/2.5.0/gems/newrelic_rpm-5.6.0.349/lib/new_relic/agent/method_tracer.rb:327:in `user_with_trace_Custom_Sinatra_Application_user'
	/edx/app/forum/cs_comments_service/api/users.rb:16:in `block in <top (required)>'
	/edx/app/forum/.gem/ruby/2.5.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1610:in `call'
	/edx/app/forum/.gem/ruby/2.5.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1610:in `block in compile!'
	/edx/app/forum/.gem/ruby/2.5.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:975:in `block (3 levels) in route!'
	/edx/app/forum/.gem/ruby/2.5.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:994:in `route_eval'
	/edx/app/forum/.gem/ruby/2.5.0/gems/newrelic_rpm-5.6.0.349/lib/new_relic/agent/instrumentation/sinatra.rb:138:in `route_eval_with_newrelic'
	/edx/app/forum/.gem/ruby/2.5.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:975:in `block (2 levels) in route!'
	/edx/app/forum/.gem/ruby/2.5.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1015:in `block in process_route'
	/edx/app/forum/.gem/ruby/2.5.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1013:in `catch'
	/edx/app/forum/.gem/ruby/2.5.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1013:in `process_route'
	/edx/app/forum/.gem/ruby/2.5.0/gems/newrelic_rpm-5.6.0.349/lib/new_relic/agent/instrumentation/sinatra.rb:118:in `process_route_with_newrelic'
	/edx/app/forum/.gem/ruby/2.5.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:973:in `block in route!'
	/edx/app/forum/.gem/ruby/2.5.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:972:in `each'
	/edx/app/forum/.gem/ruby/2.5.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:972:in `route!'
	/edx/app/forum/.gem/ruby/2.5.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1085:in `block in dispatch!'
	/edx/app/forum/.gem/ruby/2.5.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1067:in `block in invoke'
	/edx/app/forum/.gem/ruby/2.5.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1067:in `catch'
	/edx/app/forum/.gem/ruby/2.5.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1067:in `invoke'
	/edx/app/forum/.gem/ruby/2.5.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1082:in `dispatch!'
	/edx/app/forum/.gem/ruby/2.5.0/gems/newrelic_rpm-5.6.0.349/lib/new_relic/agent/instrumentation/sinatra.rb:163:in `dispatch_and_notice_errors_with_newrelic'
	/edx/app/forum/.gem/ruby/2.5.0/gems/newrelic_rpm-5.6.0.349/lib/new_relic/agent/instrumentation/sinatra.rb:149:in `block in dispatch_with_newrelic'
	/edx/app/forum/.gem/ruby/2.5.0/gems/newrelic_rpm-5.6.0.349/lib/new_relic/agent/instrumentation/controller_instrumentation.rb:370:in `perform_action_with_newrelic_trace'
	/edx/app/forum/.gem/ruby/2.5.0/gems/newrelic_rpm-5.6.0.349/lib/new_relic/agent/instrumentation/sinatra.rb:146:in `dispatch_with_newrelic'
	/edx/app/forum/.gem/ruby/2.5.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:907:in `block in call!'
	/edx/app/forum/.gem/ruby/2.5.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1067:in `block in invoke'
	/edx/app/forum/.gem/ruby/2.5.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1067:in `catch'
	/edx/app/forum/.gem/ruby/2.5.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1067:in `invoke'
	/edx/app/forum/.gem/ruby/2.5.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:907:in `call!'
	/edx/app/forum/.gem/ruby/2.5.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:895:in `call'
	/edx/app/forum/.gem/ruby/2.5.0/gems/newrelic_rpm-5.6.0.349/lib/new_relic/rack/agent_hooks.rb:30:in `traced_call'
	/edx/app/forum/.gem/ruby/2.5.0/gems/newrelic_rpm-5.6.0.349/lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
	/edx/app/forum/.gem/ruby/2.5.0/gems/newrelic_rpm-5.6.0.349/lib/new_relic/rack/browser_monitoring.rb:32:in `traced_call'
	/edx/app/forum/.gem/ruby/2.5.0/gems/newrelic_rpm-5.6.0.349/lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
	/edx/app/forum/.gem/ruby/2.5.0/bundler/gems/rack-contrib-6ff3ca2b2d98/lib/rack/contrib/locale.rb:15:in `call'
	/edx/app/forum/.gem/ruby/2.5.0/gems/newrelic_rpm-5.6.0.349/lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
	/edx/app/forum/.gem/ruby/2.5.0/gems/rack-protection-1.5.3/lib/rack/protection/xss_header.rb:18:in `call'
	/edx/app/forum/.gem/ruby/2.5.0/gems/newrelic_rpm-5.6.0.349/lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
	/edx/app/forum/.gem/ruby/2.5.0/gems/rack-protection-1.5.3/lib/rack/protection/path_traversal.rb:16:in `call'
	/edx/app/forum/.gem/ruby/2.5.0/gems/newrelic_rpm-5.6.0.349/lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
	/edx/app/forum/.gem/ruby/2.5.0/gems/rack-protection-1.5.3/lib/rack/protection/json_csrf.rb:18:in `call'
	/edx/app/forum/.gem/ruby/2.5.0/gems/newrelic_rpm-5.6.0.349/lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
	/edx/app/forum/.gem/ruby/2.5.0/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call'
	/edx/app/forum/.gem/ruby/2.5.0/gems/newrelic_rpm-5.6.0.349/lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
	/edx/app/forum/.gem/ruby/2.5.0/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call'
	/edx/app/forum/.gem/ruby/2.5.0/gems/newrelic_rpm-5.6.0.349/lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
	/edx/app/forum/.gem/ruby/2.5.0/gems/rack-protection-1.5.3/lib/rack/protection/frame_options.rb:31:in `call'
	/edx/app/forum/.gem/ruby/2.5.0/gems/newrelic_rpm-5.6.0.349/lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
	/edx/app/forum/.gem/ruby/2.5.0/gems/rack-1.6.8/lib/rack/logger.rb:15:in `call'
	/edx/app/forum/.gem/ruby/2.5.0/gems/newrelic_rpm-5.6.0.349/lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
	/edx/app/forum/.gem/ruby/2.5.0/gems/rack-1.6.8/lib/rack/commonlogger.rb:33:in `call'
	/edx/app/forum/.gem/ruby/2.5.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:219:in `call'
	/edx/app/forum/.gem/ruby/2.5.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:212:in `call'
	/edx/app/forum/.gem/ruby/2.5.0/gems/newrelic_rpm-5.6.0.349/lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
	/edx/app/forum/.gem/ruby/2.5.0/gems/rack-1.6.8/lib/rack/head.rb:13:in `call'
	/edx/app/forum/.gem/ruby/2.5.0/gems/newrelic_rpm-5.6.0.349/lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
	/edx/app/forum/.gem/ruby/2.5.0/gems/rack-1.6.8/lib/rack/methodoverride.rb:22:in `call'
	/edx/app/forum/.gem/ruby/2.5.0/gems/newrelic_rpm-5.6.0.349/lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
	/edx/app/forum/.gem/ruby/2.5.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:182:in `call'
	/edx/app/forum/.gem/ruby/2.5.0/gems/newrelic_rpm-5.6.0.349/lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
	/edx/app/forum/.gem/ruby/2.5.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:2013:in `call'
	/edx/app/forum/.gem/ruby/2.5.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1487:in `block in call'
	/edx/app/forum/.gem/ruby/2.5.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1787:in `synchronize'
	/edx/app/forum/.gem/ruby/2.5.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1487:in `call'
	/edx/app/forum/.gem/ruby/2.5.0/gems/newrelic_rpm-5.6.0.349/lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
	/edx/app/forum/.gem/ruby/2.5.0/gems/mongoid-7.0.5/lib/mongoid/query_cache.rb:118:in `block in call'
	/edx/app/forum/.gem/ruby/2.5.0/gems/mongoid-7.0.5/lib/mongoid/query_cache.rb:69:in `cache'
	/edx/app/forum/.gem/ruby/2.5.0/gems/mongoid-7.0.5/lib/mongoid/query_cache.rb:118:in `call'
	/edx/app/forum/.gem/ruby/2.5.0/gems/newrelic_rpm-5.6.0.349/lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
	/edx/app/forum/.gem/ruby/2.5.0/gems/rack-timeout-0.4.2/lib/rack/timeout/core.rb:122:in `block in call'
	/edx/app/forum/.gem/ruby/2.5.0/gems/rack-timeout-0.4.2/lib/rack/timeout/support/timeout.rb:19:in `timeout'
	/edx/app/forum/.gem/ruby/2.5.0/gems/rack-timeout-0.4.2/lib/rack/timeout/core.rb:121:in `call'
	/edx/app/forum/.gem/ruby/2.5.0/gems/newrelic_rpm-5.6.0.349/lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
	/edx/app/forum/.gem/ruby/2.5.0/gems/unicorn-5.3.1/lib/unicorn/http_server.rb:606:in `process_client'
	/edx/app/forum/.gem/ruby/2.5.0/gems/unicorn-5.3.1/lib/unicorn/http_server.rb:702:in `worker_loop'
	/edx/app/forum/.gem/ruby/2.5.0/gems/unicorn-5.3.1/lib/unicorn/http_server.rb:549:in `spawn_missing_workers'
	/edx/app/forum/.gem/ruby/2.5.0/gems/unicorn-5.3.1/lib/unicorn/http_server.rb:142:in `start'
	/edx/app/forum/.gem/ruby/2.5.0/gems/unicorn-5.3.1/bin/unicorn:126:in `<top (required)>'
	/edx/app/forum/cs_comments_service/bin/unicorn:16:in `load'
	/edx/app/forum/cs_comments_service/bin/unicorn:16:in `<main>'
I, [2020-03-18T04:17:46.804355 #2435]  INFO -- : source=rack-timeout id=517507b5a94017882261f7322061cdec timeout=20000ms service=63ms state=completed

Could this have something to do with the mongo upgrade and related code changes? @giovannicimolin @kshitij any ideas? Feel free to log time on SE-2307

Update Reported this on the community forum: Discussion forum errors in latest master with mongo 3.2

kshitij · March 18, 2020, 6:21am

I remember @guruprasad facing a similar issue. I think it was solved by setting a null value explicitly to blank or similar.

jill · March 22, 2020, 8:57am

These issues seem to have something to do with the many attempts and reverts when upgrading the mongodb ruby driver on the forum master code, so basically, the forum master code is in chaos at the moment.

Luckily, edx-platform master still works with the ironwood.2 forum code, so have updated our watched fork to use this instead:

# Jill’s fix from 2020-03-02
# adds change from https://github.com/edx/configuration/pull/5649
# forum_ruby_version: "2.5.7"
# Update 2020-03-22 issues with cs_comments_service mean we can't use latest master,
# so removed above change, and added this:
forum_version: open-release/ironwood.2

And that seems to be working ok (for now).

guruprasad · March 22, 2020, 7:07pm

@jill, @kshitij, we (@giovannicimolin and I) had fixed the forum MongoDB authentication issues by setting the FORUM_MONGO_AUTH_DB to the name of the forum database itself (used as the value of FORUM_MONGO_DATABASE) when we ran into similar errors on the Rue89 instance with mLab. I am not sure if the issue is the same as I haven’t checked the logs yet.

jill · March 23, 2020, 2:40am

Oo that’s interesting… I’m going to leave the forum version at ironwood.2 for new PR sandboxes until we have further issues with the forum, since there is still a lot of back-and-forth merging and reverting happening on the master branch there. But will try that next, thank you!

guruprasad · April 3, 2020, 6:12pm

Some edx/configuration PRs of interest:

The https://github.com/edx/configuration/pull/5691 PR is renaming the *_version ansible variables to upper case. Once it is merged, this will likely break our master sandboxes and will require Ocim changes for Juniper (CC @toxinu).

The https://github.com/edx/configuration/pull/5718 PR has added an Ansible code conventions document and created a directory for ADRs.

jill · April 17, 2020, 4:19am

Thank’s to Bill DeRusha’s merged configuration#5723, I’ve been able to remove some things from our master PR watched fork configuration.

Notes:

In the interests of keeping our PR sandbox builds stable, I decided to continue using our ocim-master configuration branch, but have updated it to match the current upstream master (9810e9817…eda5f35a5).
The EDXAPP_VIDEO_IMAGE_SETTINGS and EDXAPP_VIDEO_TRANSCRIPTS_SETTINGS either need an upstream PR, or an Ocim change to apply them to all our instances.
EDIT ~~The forum_version has been bumped to open-release/juniper.alpha1, since there’s still~~We’ve merged PRs to resolve the issues with our mongo and master with upstream PRs, and will be addressing the authentication mechanism update in the follow-up tasks for SE-2383. I’ve updated the watched fork and notes below accordingly.

FYI @usman @braden for LabXchange CI.

name	value
configuration_source_repo_url	`https://github.com/open-craft/configuration`
configuration_version	`ocim-master`
configuration_playbook_name	playbooks/openedx_native.yml
edx_platform_repository_url	`https://github.com/edx/edx-platform`
edx_platform_commit	`release`
openedx_release	`master`
configuration_extra_settings	# Braden’s other temporary fix 2020-01-23 # https://chat.opencraft.com/opencraft/pl/4ckt5atqkbyzbmddcx8q93bw8h EDXAPP_VIDEO_IMAGE_SETTINGS: VIDEO_IMAGE_MAX_BYTES : 2097152 VIDEO_IMAGE_MIN_BYTES : 2048 STORAGE_KWARGS: location: “video-images/” bucket: “{{ EDXAPP_AWS_STORAGE_BUCKET_NAME }}” # https://chat.opencraft.com/opencraft/pl/4ckt5atqkbyzbmddcx8q93bw8h EDXAPP_VIDEO_TRANSCRIPTS_SETTINGS: VIDEO_TRANSCRIPTS_MAX_BYTES : 3145728 STORAGE_KWARGS: location: “video-transcripts/” bucket: “{{ EDXAPP_AWS_STORAGE_BUCKET_NAME }}” # Update 2020-04-17 issues with forum mongo driver and our authentication mechanism: # https://discuss.openedx.org/t/discussion-forum-errors-in-latest-master-with-mongo-3-2/1843 FORUM_MONGO_AUTH_MECH: “:mongodb_cr” # Sane worker settings EDXAPP_WORKERS: lms: 3 cms: 2

Click to see diff

--- config.old	2020-04-17 13:34:56.000000000 +0930
+++ config.now	2020-04-17 13:34:49.000000000 +0930
@@ -1,17 +1,3 @@
-##########################################################
-# See https://forum.opencraft.com/t/open-edx-devstack-alerts/515/7
-#
-# -> configuration version pinned to open-craft:ocim-master instead of ‘edx:master’
-# Includes:
-# * Braden’s temporary fix 2020-01-29 to work around https://github.com/edx/configuration/pull/5414#issuecomment-579997504 (5806881b0071aeef355054e7b658855fc22e4035)
-# * Jill's fix 2020-03-13
-#    https://github.com/edx/configuration/pull/5635
-#    Removal of Django Auth Provider and create_oauth2_client from edx-platform
-# 
-# Braden’s temporary fix for configuration bug - remove when edX fixes it or removes the prospectus code https://github.com/edx/configuration/pull/5561
-PROSPECTUS_PREVIEW_NGINX_USERS: []
-PROSPECTUS_PREVIEW_HTPASSWD_PASS: "doesntmatter!"
-PROSPECTUS_PREVIEW_HTPASSWD_USER: "dontuse"
 # Braden’s other temporary fix 2020-01-23
 # https://chat.opencraft.com/opencraft/pl/4ckt5atqkbyzbmddcx8q93bw8h
 EDXAPP_VIDEO_IMAGE_SETTINGS:
@@ -27,23 +13,11 @@
   location: "video-transcripts/"
   bucket: "{{ EDXAPP_AWS_STORAGE_BUCKET_NAME }}"
 
-# Jill’s fix from 2020-03-02
-# adds change from https://github.com/edx/configuration/pull/5649
-# forum_ruby_version: "2.5.7"
-# Update 2020-03-22 issues with cs_comments_service mean we can't use latest master,
-# so removed above change, and added this:
-forum_version: open-release/ironwood.2
-
-# Allows Studio and LMS to share session cookies, so lets Studio login via SSO to LMS
-EDXAPP_SESSION_COOKIE_DOMAIN: ".{{ EDXAPP_LMS_BASE }}"
-##########################
-
-# Other fixes
-ICP_LICENSE_INFO: {}
-AWS_GATHER_FACTS: false
+# Update 2020-04-17 issues with forum mongo driver and our authentication mechanism:
+# https://discuss.openedx.org/t/discussion-forum-errors-in-latest-master-with-mongo-3-2/1843
+FORUM_MONGO_AUTH_MECH: ":mongodb_cr"
 
 # Sane worker settings
 EDXAPP_WORKERS:
   lms: 3
   cms: 2
-COMMON_RETIREMENT_SERVICE_SETUP: false