Changing config on LMS app servers without downtime

Say you’re fighting a fire and you urgently need to change some code or configuration (django setting) for the LMS on a running appserver (you don’t have time to wait for a new ansible deployment).

Once you’ve made the configuration change or edited the LMS code, you can tell gunicorn to restart the LMS processes gracefully, using kill -HUP <PID> with the PID of the overall gunicorn process that controls the LMS workers.

This should apply your fix without any downtime for learners.

6 Likes

Great tip, thank you @braden !

Interesting, thanks for the tip @braden :) I am just wondering how it works behind the scene, or if supervisorctl can be used to write a custom command to reload the config :slight_smile: Probably we can upstream that too.

I imagine that’s possible yes. In fact it may even exist and we don’t know about it?

It’s possible to make it a one-liner:

kill -HUP $(/edx/bin/supervisorctl pid lms)

or

kill -HUP $(ps -eopid,cmd | grep lms_gunicorn | grep -vm1 grep | awk '{ print $1 }')
2 Likes

This is nice!

On a related note: edX has a tool to automatically update configurations on all appservers/containers called Hermes (GitHub - edx/hermes: Hermes is the paper pusher/messenger for edx config files) - it downloads and decrypts the new configuration and restarts the LMS/Studio/others services.

I’ve bumped into it some time ago during the BD projects. IIRC edX has a repo with all configuration files (called remote-configuration or something like that) and any changes made to it are somehow picked up by Hermes and deployed to all appservers.

2 Likes