@josh Thanks for the script and idea! You’re brave trying to tame the devstack in this way.
You can also keep DB snapshots with ./dump-db edxapp and load-db but I don’t know how well they work. After you have the Hawthorn DB you can keep dumps.
Another idea if Josh’s script doesn’t work would be: instead of going master→hawthorn backwards, drop the DB and go 0→hawthorn forwards. But this surely takes more time.
Nice work @josh! 
When working on separate devstacks say master and hawthorn, I see that the backend data is shared because the MySQL, MongoDB containers use the same name and the same volumes. So this is very useful.
A quick tip for those who aren’t using this yet. You can create a file called local.mk in your devstack directory, and it will be imported by the main Makefile. Here are some of the things I have in my local.mk:
# When working on React components for edx-platform this can save time vs a full
# make lms-static
lms-webpack:
docker exec -t edx.devstack.lms bash -c \
'source /edx/app/edxapp/edxapp_env && \
cd /edx/app/edxapp/edx-platform/ && \
paver webpack'
# Copies over generated reports, like the problem responses report to a reports
# directory in the devstack.
get-reports:
docker cp edx.devstack.lms:/tmp/edx-s3/ reports/
# Commits all containers. Can't remember what I used this for. I don't anymore.
commit-all:
@docker ps -f name=edx.devstack --format "{{.ID}} {{.Image}}" | grep edxops | while read command; do \
docker commit $$command; \
done
# Run a test by providing a path, run as make run-test TEST=lms/djangoapps/...
run-test:
@docker exec -it edx.devstack.lms bash -c \
'source /edx/app/edxapp/edxapp_env && \
cd /edx/app/edxapp/edx-platform/ && \
pytest $(TEST) $(PARAMS)'
# I like the micro text editor, and this just installs that into the devstacks
install-micro:
@docker cp /usr/local/bin/micro edx.devstack.lms:/usr/local/bin/micro
@docker cp /usr/local/bin/micro edx.devstack.studio:/usr/local/bin/micro
# This allows you to install an XBlock in both lms and studio using
# make install-xblock XBLOCK=problem-builder
install-xblock:
for c in lms studio ; do \
docker exec -it edx.devstack.$$c bash -c \
'cd /edx/src/xblocks/$(XBLOCK) && /edx/app/edxapp/venvs/edxapp/bin/pip install -e .' ; \
done
# Takes you directly to the LMS python console
lms-python:
docker exec -it edx.devstack.lms env TERM=$(TERM) /edx/bin/edxapp-shell-lms
# Restart both containers
edx-restart: lms-restart studio-restart
Another thing I find useful, I have a shell function called edx that runs make -C $HOME/projects/devstack $@
This allows me to run devstack commands from any directory as edx run-test ... or edx dev.up etc.
@kshitij, these are very useful. 
@kshitij, I use something similar where the path to the devstack directory is set by a folder-specific environment variable loaded by direnv and the shell function uses it.
If anyone gets the issue No module named tz while browsing the courseware, it’s a problem with stale cache. To fix it, you just need to restart the memcached container on the Docker devstack:
docker-compose restart memcached
More info here: https://openedx.atlassian.net/wiki/spaces/OXA/pages/926286179/Devstack+No+module+named+tz
I’m used to being able to easily snapshot and restore btrfs-backed LXC devstacks, including assets and database data. This is particularly useful when dealing with master, as it will sometimes get broken upstream, leaving your own work blocked while the issue isn’t fixed. Or, you may just want to restore things quickly to a known working state after wreaking havoc in the database, without having to start from scratch.
It turns out this is a little more complicated with the Docker devstack than an lxc snapshot or lxc restore. I did my best to replicate the behavior in this devstack branch (HEAD at time of writing), so that a make snapshot and a make snapshot-restore does the right thing.
It is currently not meant for general consumption, as volume snapshots only work on btrfs-mounted docker roots. But a little tweaking would get it to work on regular filesystems (though it would be slower and take up more space - hence why btrfs rlz 
).
That’s excellent! I’m going back to the docker devstack after six months almost exclusively on the Vagrant one, and I must say the ability of taking snapshots really make a difference in productivity. I would take a snapshot, test deletion of users, companies, check the results and then restore the snapshot.
I’m actually testing another approach using LVM and the docker devstack, having created a 60GB logical volume and mounted that in /var/lib/docker. I still haven’t tested it properly and want to create a script/Makefile to handle backing and restoring as there are some specific steps for each, like stopping containers etc.
Let us know how your implementation goes, I’ll do the same whenever I have the chance to properly test the snapshotting features.
This is precisely what I missed. Plus, in addition to random pypi/github failures, my download speeds aren’t fiber-fabulous, so rebuilding containers always takes a while. This will end up saving a bunch of time.
Take a look at what I did. Might save you some time. Though, of course, if you’re snapshotting /var/lib/docker all at once, all that parsing won’t be necessary.
Awesome, please do!
Great experimentation!
I saw that the devstack contains some scripts to take and restore snapshots, though I didn’t try them.
Note that to capture the state of the devstack you’ll also need to snapshot the code in all the folders that are shared with the container: edx-platform, etc.
Hadn’t seen those, thanks for the pointer! However, they take a different approach, being more geared towards offline distribution of the output versus quick snapshotting and restoring. I could, however, take some inspiration and rewrite the snapshot script in python to make it actually readable, haha!
Correct. Though since all of that is in git and better managed manually anyway, I would prefer a snapshot/restore script to not mess with the code I’m working on. (Which is how I handle LXC/KVM devstack mounts anyway.)
edX have added new python3 tests added to the edx-platform CI, and they are causing some pain for some of our PRs. Feanil shared how to run the python3 python tests, since this is unfortunately absent from the edx-platform testing docs:
tox -e py35-django111 -- pytest /path/to/test.py
I assume something similar will work for the quality tests too.
And to trigger the specific test in the CI, put this in a comment on your PR:
jenkins run py35-django111 <context>
e.g.
jenkins run py35-django111 python
Most of our devstacks require setting a shell environment variable (or two) to get working properly. For example,
export OPENEDX_RELEASE=ironwood.master
It can be a hassle to make sure the right environment is set if you move between devstacks. One nice tool for managing this is direnv. If I have direnv installed, I can add a .envrc to each devstack directory to ensure the correct evironment is set when I enter the directory, and unset when
I leave it. For example,
# ~/OpenCraft/devstack-ironwood/devstack/.envrc
export OPENEDX_RELEASE=ironwood.master
export VIRTUAL_ENV=venv
layout python-venv
This ensures that when I enter ~/OpenCraft/devstack-ironwood/devstack/ OPENEDX_RELEASE and VIRTUAL_ENV are set appropriately. The last line also ensures the python virtual environment has been set up (which is a bonus). Checkout the direnv for appropriate shell integration and options for different types of virtual environments.
When I enter a directory controlled by direnv, I get an informative message:
~ $ cd ~/OpenCraft/devstack-ironwood/devstack
direnv: loading .envrc
direnv: export +OPENEDX_RELEASE +VIRTUAL_ENV ~PATH
(venv) ~/OpenCraft/devstack-ironwood/devstack $
(venv) ~ $ echo $OPENEDX_RELEASE
ironwood.master
(venv) ~ $
Similarly, when I leave:
(venv) ~/OpenCraft/devstack-ironwood/devstack $ cd ~
direnv: unloading
~ $ echo $OPENEDX_RELEASE
~$
In case you are worried about the security of this, note that adding or editing .envrc files requires approval:
$ vim .envrc
direnv: error .envrc is blocked. Run `direnv allow` to approve its content.
$
direnv is news to me. Nice! Thanks, @kahlil, this will be usefull for a bunch of things!
In each top-level devstack directory, I usually set up a .envrc file with the appropriate environment variables and a variable EDX_DEVSTACK_DIR which points to the devstack repo folder within that top-level folder.
Then with a snippet like
edx() {
if [ ! -z "$EDX_DEVSTACK_DIR" ]; then
make -C "$EDX_DEVSTACK_DIR" "$@"
fi
}
I can run all the docker devstack Makefile commands from any directory inside the top-level devstack directory like edx <command>, without having to be in the <devstack repo dir> itself. This becomes super-handy when one also adds a lot of custom targets in <devstack repo dir>/local.mk. I have custom targets for updating the LMS configuration, updating assets, just compiling SASS, setting SiteConfiguration variables etc.
I have all the directories and files related to work under ~/d/opencraft, where I have a top-level .envrc file which sets the environment variables for the git author and committer. All the other .envrc files within sub-directories of that directory, use the source_up function to load the configuration in the top-level directory as well. With this, I don’t have to set the git committer/author details for every repository or set it globally (which will have to be overridden for non-work repositories).
Thanks @guruprasad! Thats a very cool multi-level setup 
. I especially like the idea of setting git author/committer environment variables. I did not know about those:
GIT_AUTHOR_NAME
GIT_AUTHOR_EMAIL
GIT_COMMITTER_NAME
GIT_COMMITTER_EMAIL
This will make it a lot easier to isolate work and non-work identities.
I just updated my (old) devstack and ran into this issue:
pymongo.errors.ServerSelectionTimeoutError: edx.devstack.mongo:27017: [Errno -2] Name or service not known
Running make mongo-logs showed this was why mongo wasn’t running:
IMPORTANT: UPGRADE PROBLEM: The data files need to be fully upgraded to version 3.4 before attempting an upgrade to 3.6; see http://dochub.mongodb.org/core/3.6-upgrade-fcv for more details.
If anyone else is in the same situation, here is the procedure for upgrading your data files without having to erase your docker volume (and all your courses). It’s actually quite easy:
MONGO_VERSION=3.4.24 make dev.up.mongo
make mongo-shell
mongo
db.adminCommand( { setFeatureCompatibilityVersion: "3.4" } )
exit
exit
docker container stop edx.devstack.mongo
make dev.up.mongo
make mongo-shell
mongo
db.adminCommand( { setFeatureCompatibilityVersion: "3.6" } )
exit
exit
edX added a script to do this: Upgrade devstack to Mongo 3.6.17 OPS-4531 #486
Came back here for this. Thanks @guruprasad and @kahlil . 
It goes very nicely with this PR to the edx Docker devstack that Kyle McCormick is reviewing: it finally allows one to have multiple coexisting devstacks via a simple export COMPOSE_PROJECT_NAME=whatever.
You still can’t run them simultaneously, but it’s nevertheless a huge improvement. I’ve already incorporated it locally. Let’s hope it gets merged soon.
Great to see this! This is a much needed improvement.
Hopefully it’s easy to backport to older releases.
The PR seems pretty straightforward, though the diff is large. A whole bunch of places to sub docker exec with docker-compose exec, etc.
In any case, heads-up if anybody’s trying this with the LabXchange devstack! You’ll need blockstore!1 as well as labxchange-dev!839.