That’s excellent! I’m going back to the docker devstack after six months almost exclusively on the Vagrant one, and I must say the ability of taking snapshots really make a difference in productivity. I would take a snapshot, test deletion of users, companies, check the results and then restore the snapshot.
I’m actually testing another approach using LVM and the docker devstack, having created a 60GB logical volume and mounted that in /var/lib/docker. I still haven’t tested it properly and want to create a script/Makefile to handle backing and restoring as there are some specific steps for each, like stopping containers etc.
Let us know how your implementation goes, I’ll do the same whenever I have the chance to properly test the snapshotting features.
This is precisely what I missed. Plus, in addition to random pypi/github failures, my download speeds aren’t fiber-fabulous, so rebuilding containers always takes a while. This will end up saving a bunch of time.
Take a look at what I did. Might save you some time. Though, of course, if you’re snapshotting /var/lib/docker all at once, all that parsing won’t be necessary.
Note that to capture the state of the devstack you’ll also need to snapshot the code in all the folders that are shared with the container: edx-platform, etc.
Hadn’t seen those, thanks for the pointer! However, they take a different approach, being more geared towards offline distribution of the output versus quick snapshotting and restoring. I could, however, take some inspiration and rewrite the snapshot script in python to make it actually readable, haha!
Correct. Though since all of that is in git and better managed manually anyway, I would prefer a snapshot/restore script to not mess with the code I’m working on. (Which is how I handle LXC/KVM devstack mounts anyway.)
Most of our devstacks require setting a shell environment variable (or two) to get working properly. For example,
export OPENEDX_RELEASE=ironwood.master
It can be a hassle to make sure the right environment is set if you move between devstacks. One nice tool for managing this is direnv. If I have direnv installed, I can add a .envrc to each devstack directory to ensure the correct evironment is set when I enter the directory, and unset when
I leave it. For example,
This ensures that when I enter ~/OpenCraft/devstack-ironwood/devstack/OPENEDX_RELEASE and VIRTUAL_ENV are set appropriately. The last line also ensures the python virtual environment has been set up (which is a bonus). Checkout the direnv for appropriate shell integration and options for different types of virtual environments.
When I enter a directory controlled by direnv, I get an informative message:
In each top-level devstack directory, I usually set up a .envrc file with the appropriate environment variables and a variable EDX_DEVSTACK_DIR which points to the devstack repo folder within that top-level folder.
Then with a snippet like
edx() {
if [ ! -z "$EDX_DEVSTACK_DIR" ]; then
make -C "$EDX_DEVSTACK_DIR" "$@"
fi
}
I can run all the docker devstack Makefile commands from any directory inside the top-level devstack directory like edx <command>, without having to be in the <devstack repo dir> itself. This becomes super-handy when one also adds a lot of custom targets in <devstack repo dir>/local.mk. I have custom targets for updating the LMS configuration, updating assets, just compiling SASS, setting SiteConfiguration variables etc.
I have all the directories and files related to work under ~/d/opencraft, where I have a top-level .envrc file which sets the environment variables for the git author and committer. All the other .envrc files within sub-directories of that directory, use the source_up function to load the configuration in the top-level directory as well. With this, I don’t have to set the git committer/author details for every repository or set it globally (which will have to be overridden for non-work repositories).
Thanks @guruprasad! Thats a very cool multi-level setup . I especially like the idea of setting git author/committer environment variables. I did not know about those:
I just updated my (old) devstack and ran into this issue:
pymongo.errors.ServerSelectionTimeoutError: edx.devstack.mongo:27017: [Errno -2] Name or service not known
Running make mongo-logs showed this was why mongo wasn’t running:
IMPORTANT: UPGRADE PROBLEM: The data files need to be fully upgraded to version 3.4 before attempting an upgrade to 3.6; see http://dochub.mongodb.org/core/3.6-upgrade-fcv for more details.
If anyone else is in the same situation, here is the procedure for upgrading your data files without having to erase your docker volume (and all your courses). It’s actually quite easy:
MONGO_VERSION=3.4.24 make dev.up.mongo
make mongo-shell
mongo
db.adminCommand( { setFeatureCompatibilityVersion: "3.4" } )
exit
exit
docker container stop edx.devstack.mongo
make dev.up.mongo
make mongo-shell
mongo
db.adminCommand( { setFeatureCompatibilityVersion: "3.6" } )
exit
exit
It goes very nicely with this PR to the edx Docker devstack that Kyle McCormick is reviewing: it finally allows one to have multiple coexisting devstacks via a simple export COMPOSE_PROJECT_NAME=whatever.
You still can’t run them simultaneously, but it’s nevertheless a huge improvement. I’ve already incorporated it locally. Let’s hope it gets merged soon.
The original PR is still not finalized, but I was able to fire up both an Ironwood and a Juniper devstack. You still can’t run them at the same time, but it’s nice to have both releases at the ready for when they’re needed.
I now have both ironwood and master up and isolated. Here is my guide for how that can be done with the new multi-devstack code.
Destroy your current devstack. Sorry about this. I don’t know a way around it.
Create a new directory for your ironwood devstack project.
Check out the ironwood branch of the devstack in it and set up a new devstack with the ironwood release set.
This devstack will use launch services as edx.devstack.*
In your master devstack create a new file called options.local.mk that has COMPOSE_PROJECT_NAME=devstack-master
Set up this devstack from scratch. Now you master devstack is ready with services at edx.devstack-master.*
If you need a juniper devstack eventually, just create a local tag for juniper right now, it doesn’t matter if edX hasn’t tagged it, you can still use it with Juniper since it’s close enough
Webpack hot reload doesn’t work for Vagrant devstack, because when you edit file in shared directory, change event doesn’t be propagated to the guest system.
Could I know where you’re using the Vagrant devstack? Your use case sounds like you’re talking about Ocim here. If that is the case, is there any particular reason you’re running the frontend inside the Vagrant devstack instead of outside it? I think that is the recommendation for Ocim. I don’t see any good reason to run the frontend dev server inside vagrant.
. I especially like the idea of setting git author/committer environment variables. I did not know about those:
