Post here any ideas/problems/solutions. Just like Mattermost, but easier to browse and to search old posts.
Or better: we can post only at https://forum.opencraft.com/c/tips-tricks and delete this thread.
Thanks for starting this @daniel
The tips and tricks is a category - I think it still makes sense to group tips&tricks specifically about devstack in a single thread. If it became so big that one thread would become messy we can revisit, but in the meantime that makes it easier to have a single one to point to.
I forgot to say: you can log time in this thread at the ticket you’re working at that’s giving you devstack problems/ideas/fixes.
In the docker devstack, after make lms-shell the lines I type are broken at a certain column (much before the right window’s edge). I think it’s solved by apt-get install ncurses-term inside the box, but maybe you found more permanent solutions.
In the docker devstack, if your changes to an .html aren’t being loaded even after refreshing, recompiling the theme, and restarting LMS , then you need to know that there’s a timer or caché somewhere and you need to wait for the next run (in my computer it’s exactly 3 minutes)… or you can run docker restart edx.devstack.memcached to make it use the new version.
I’ve found that as soon as I resize my window, it works fine. (edit: I don’t consider this a good solution, fwiw.)
I found that even with ncurses-term installed, it would still sometimes be fixed at 80x24. I added a shell alias in my host environment:
alias lmsh="docker exec -it edx.devstack.lms bash -c \"stty cols $COLUMNS rows $LINES && /edx/app/edxapp/devstack.sh open\""
I made a similar alias for Studio as well. This could probably be added in a more generic Makefile target, but I was out of patience with Docker, so this is what I’ve been using.
Also, even with this in place, readline redisplay, as when going back through history at the shell or in a Python interpreter, was broken for me, so I was constantly having to redisplay with control-l to avoid executing the wrong thing. Editing the default PS1 in .bashrc inside the container to be simply PS1='\u@\h:\w\$ ' corrects this. 
Both of these fixes were found in GitHub issues or Stack Overflow, I think. Can’t take credit.
One more thing about Docker shells: if you’re used to Emacs keybindings and want to use Control-P to go back through history, instead of arrow keys, and without hitting it twice for every line, you have to rebind your detach sequence, either in the docker command with --detachKeys or in your ~/.docker/config.json.
The default make lms-shell is:
docker exec -it edx.devstack.lms env TERM=xterm-256color \
/edx/app/edxapp/devstack.sh open
I’m currently using
docker exec -it edx.devstack.lms env TERM=xterm-256color \
LINES=$LINES \
COLUMNS=$COLUMNS \
/edx/app/edxapp/devstack.sh open
and it seems to work pretty well.
When running vagrant based devstack, be careful not to run:
apt-key adv --keyserver keys.gnupg.net --recv-keys 69464050
but make sure you are running:
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 69464050
1 Like
Calling make dev.provision fails, at random points, now at:
Creating indexes for edx_notes_api...
Traceback (most recent call last):
File "manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "/edx/app/edx_notes_api/venvs/edx_notes_api/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
utility.execute()
File "/edx/app/edx_notes_api/venvs/edx_notes_api/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 356, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/edx/app/edx_notes_api/venvs/edx_notes_api/local/lib/python2.7/site-packages/django/core/management/base.py", line 283, in run_from_argv
self.execute(*args, **cmd_options)
File "/edx/app/edx_notes_api/venvs/edx_notes_api/local/lib/python2.7/site-packages/django/core/management/base.py", line 327, in execute
self.check()
File "/edx/app/edx_notes_api/venvs/edx_notes_api/local/lib/python2.7/site-packages/django/core/management/base.py", line 359, in check
include_deployment_checks=include_deployment_checks,
File "/edx/app/edx_notes_api/venvs/edx_notes_api/local/lib/python2.7/site-packages/django/core/management/base.py", line 346, in _run_checks
return checks.run_checks(**kwargs)
File "/edx/app/edx_notes_api/venvs/edx_notes_api/local/lib/python2.7/site-packages/django/core/checks/registry.py", line 81, in run_checks
new_errors = check(app_configs=app_configs)
File "/edx/app/edx_notes_api/venvs/edx_notes_api/local/lib/python2.7/site-packages/django/core/checks/model_checks.py", line 30, in check_all_models
errors.extend(model.check(**kwargs))
File "/edx/app/edx_notes_api/venvs/edx_notes_api/local/lib/python2.7/site-packages/django/db/models/base.py", line 1284, in check
errors.extend(cls._check_fields(**kwargs))
File "/edx/app/edx_notes_api/venvs/edx_notes_api/local/lib/python2.7/site-packages/django/db/models/base.py", line 1359, in _check_fields
errors.extend(field.check(**kwargs))
File "/edx/app/edx_notes_api/venvs/edx_notes_api/local/lib/python2.7/site-packages/django/db/models/fields/__init__.py", line 913, in check
errors = super(AutoField, self).check(**kwargs)
File "/edx/app/edx_notes_api/venvs/edx_notes_api/local/lib/python2.7/site-packages/django/db/models/fields/__init__.py", line 219, in check
errors.extend(self._check_backend_specific_checks(**kwargs))
File "/edx/app/edx_notes_api/venvs/edx_notes_api/local/lib/python2.7/site-packages/django/db/models/fields/__init__.py", line 322, in _check_backend_specific_checks
return connections[db].validation.check_field(self, **kwargs)
File "/edx/app/edx_notes_api/venvs/edx_notes_api/local/lib/python2.7/site-packages/django/db/backends/mysql/validation.py", line 49, in check_field
field_type = field.db_type(self.connection)
File "/edx/app/edx_notes_api/venvs/edx_notes_api/local/lib/python2.7/site-packages/django/db/models/fields/__init__.py", line 644, in db_type
return connection.data_types[self.get_internal_type()] % data
File "/edx/app/edx_notes_api/venvs/edx_notes_api/local/lib/python2.7/site-packages/django/utils/functional.py", line 35, in __get__
res = instance.__dict__[self.name] = self.func(instance)
File "/edx/app/edx_notes_api/venvs/edx_notes_api/local/lib/python2.7/site-packages/django/db/backends/mysql/base.py", line 174, in data_types
if self.features.supports_microsecond_precision:
File "/edx/app/edx_notes_api/venvs/edx_notes_api/local/lib/python2.7/site-packages/django/utils/functional.py", line 35, in __get__
res = instance.__dict__[self.name] = self.func(instance)
File "/edx/app/edx_notes_api/venvs/edx_notes_api/local/lib/python2.7/site-packages/django/db/backends/mysql/features.py", line 53, in supports_microsecond_precision
return self.connection.mysql_version >= (5, 6, 4) and Database.version_info >= (1, 2, 5)
File "/edx/app/edx_notes_api/venvs/edx_notes_api/local/lib/python2.7/site-packages/django/utils/functional.py", line 35, in __get__
res = instance.__dict__[self.name] = self.func(instance)
File "/edx/app/edx_notes_api/venvs/edx_notes_api/local/lib/python2.7/site-packages/django/db/backends/mysql/base.py", line 385, in mysql_version
with self.temporary_connection() as cursor:
File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
return self.gen.next()
File "/edx/app/edx_notes_api/venvs/edx_notes_api/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 591, in temporary_connection
cursor = self.cursor()
File "/edx/app/edx_notes_api/venvs/edx_notes_api/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 254, in cursor
return self._cursor()
File "/edx/app/edx_notes_api/venvs/edx_notes_api/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 229, in _cursor
self.ensure_connection()
File "/edx/app/edx_notes_api/venvs/edx_notes_api/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 213, in ensure_connection
self.connect()
File "/edx/app/edx_notes_api/venvs/edx_notes_api/local/lib/python2.7/site-packages/django/db/utils.py", line 94, in __exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/edx/app/edx_notes_api/venvs/edx_notes_api/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 213, in ensure_connection
self.connect()
File "/edx/app/edx_notes_api/venvs/edx_notes_api/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 189, in connect
self.connection = self.get_new_connection(conn_params)
File "/edx/app/edx_notes_api/venvs/edx_notes_api/local/lib/python2.7/site-packages/django/db/backends/mysql/base.py", line 274, in get_new_connection
conn = Database.connect(**conn_params)
File "/edx/app/edx_notes_api/venvs/edx_notes_api/local/lib/python2.7/site-packages/MySQLdb/__init__.py", line 81, in Connect
return Connection(*args, **kwargs)
File "/edx/app/edx_notes_api/venvs/edx_notes_api/local/lib/python2.7/site-packages/MySQLdb/connections.py", line 193, in __init__
super(Connection, self).__init__(*args, **kwargs2)
django.db.utils.OperationalError: (2005, "Unknown MySQL server host 'edx.devstack.mysql' (0)")
make: *** [dev.provision.run] Error 1
Now under Linux:
TASK [demo : import demo course] ***********************************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: OSError: [Errno 13] Permission denied: '/edx/app/e
dxapp/edx-platform'
fatal: [127.0.0.1]: FAILED! => {"changed": false, "failed": true, "module_stderr": "Traceback (most recent call last):\n File \"/tmp/ansible_aozpF
2/ansible_module_command.py\", line 212, in <module>\n main()\n File \"/tmp/ansible_aozpF2/ansible_module_command.py\", line 155, in main\n
os.chdir(chdir)\nOSError: [Errno 13] Permission denied: '/edx/app/edxapp/edx-platform'\n", "module_stdout": "", "msg": "MODULE FAILURE"}
to retry, use: --limit @/edx/app/edx_ansible/edx_ansible/playbooks/demo.retry
I’ve had some issues setting up my local Vagrant devstack. Issues:
- The Virtualbox Guest Additions where not compatible, so, I had a ton of problems when mounting the folders. I’ve solved that this way:
- Downloaded the ginkgo vagrant box manually
- Ran it and installed the correct VBoxGuestAdditions
- [Other fixes]
- Created a new Vagrant box with fixes
- Set up the Vagrantfile to use that box
-
I didn’t had any write permissions on the mounted folders inside vagrant. I solved that by disabling NFS support with
export VAGRANT_USE_VBOXFS=true, but your results may vary on this. -
When mounting the shared folders, the Vagrantfile tried to use a
ecommerce_workerto give ownership to it’s respective folder. However, this user doesn’t exist, but the fix for that is simple: in the Vagrant box, the correct username isecomworker, so just replace the line
:ecommerce_worker => {:repo => "ecommerce-worker", :local => "/edx/app/ecommerce_worker/ecommerce_worker", :owner => "ecommece_worker"},
With
:ecommerce_worker => {:repo => "ecommerce-worker", :local => "/edx/app/ecommerce_worker/ecommerce_worker", :owner => "ecomworker"},
That worked for me. -
The ansible-playbooks were failing because of an expired key when trying to update the system, and to fix that, I’ve just ran this command in the section [Other fixes] from issue 1 (that way I created a local image with most of the fixes):
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 69464050 -
Bower repository problems: the url that bower tried to used to install packages was depreacated. Fixed that with a script that ran before the original provisioning script (script below).
-
Permissions problems on python venv, also fixed with the script below.
For issues 5 and 6, I’ve modified the Vagrantfile to run a script before starting to run the ansible-playbooks:
Set up the script just above the original one:
$fixup = <<SCRIPT
#!/bin/bash
set -eux
echo '{"registry": "https://registry.bower.io"}' > /.bowerrc
if [ -d "/edx/app/edx_ansible/venvs/edx_ansible/lib/python2.7/site-packages" ]; then
chown edx-ansible:edx-ansible /edx/app/edx_ansible/venvs/edx_ansible/lib/python2.7/site-packages -R
fi
SCRIPT
And just before config.vm.provision "shell", inline: $script, I’ve inserted:
config.vm.provision "shell", inline: $fixup
- Another problem you might run into is that the MongoDB lock with an error and locks he ansible-playbook in the middle of the process. To fix that:
- When the ansible-playbook states that it’s waiting for MongoDB to come online run
vagrant sshon another command line - Run the following script as root:
rm /edx/var/mongo/mongodb/mongod.lock
mongod -repair --config /etc/mongod.conf
chown -R mongodb:mongodb /edx/var/mongo/.
service mongod start
- As soon as it finishes, the ansible-playbook will continue
After that I didn’t had any other problems. I’ve set up the devstack some 3 times using this fixes.
Many thanks to Guruprasad and Daniel for helping me setting up my devstack. 
2 Likes
@giovannicimolin, nice work documenting the issues you encountered and the steps to fix them 
1 Like
Thanks for this! It’s really helpful for anyone working on the Vagrant/Solutions devstack. I’d recommend linking any newcomer here since it seems to solve a lot of the problems they are likely to face.
One point of note though. VBOXFS is pretty slow compared to NFS, though it’s much simpler to set up.
I’ve also changed my Vagrantfile to not keep altering the exports file. After setting up NFS once I have changed the NFS config in my Vagrantfile to:
config.vm.synced_folder v[:repo], v[:local],
create: true,
type: "nfs",
nfs_version: 4,
nfs_udp: false,
mount_options: ['actimeo=2', 'tcp'],
nfs_export: false,
verify_installed: false
This gets rid of the sudo requirement, and just generally seems to work well for me, unless I update my system and don’t reboot.
1 Like
Thank you for the tips @giovannicimolin
I’ve had problems mounting the shared folders as well. Something wrong whiling adding the kernel modules. Running vagrant reload --provision after the faulty vagrant exit made it work.
I also used export VAGRANT_USE_VBOXFS=true and changed the ecommerce_worker used as you pointed out (thanks again 
).
The weird thing is that in the Vagrantfile the vbguest plugin is marked to auto reboot (config.vbguest.auto_reboot = true), but apparently is not rebooting at all. There’s also an issue open here
Edit: I also had to run the fixes for the steps 5 and 6.
1 Like
Great documentation! Thanks!
I also added this to the beginning of the $script because apt-get upgrade was failing for me:
sudo apt-key update
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv 69464050
That should automatically update the necessary keys.
Also, I moved my Vagrantfile to another folder and then the NFS weren’t mounting correctly. I noticed it was trying to mount the old paths. To fix that you have to delete /etc/exports. Vagrant lists there the NFS mounts, apparently.
I had an issue with the rabbitmq repo during apt-get update.
Err:19 https://packagecloud.io/rabbitmq/rabbitmq-server/ubuntu xenial InRelease
The following signatures couldn't be verified because the public key is not available: NO_PUBKEY F6609E60DC62814E
Fetched 27.0 kB in 3s (8,233 B/s)
Reading package lists... Done
W: http://packages.elastic.co/elasticsearch/1.5/debian/dists/stable/Release.gpg: Signature by key 46095ACC8548582C1A2699A9D27D666CD88E42B4 uses weak digest algorithm (SHA1)
W: http://downloads-distro.mongodb.org/repo/ubuntu-upstart/dists/dist/Release.gpg: Signature by key 492EAFE8CD016A07919F1D2B9ECBEC467F0CEB10 uses weak digest algorithm (SHA1)
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://packagecloud.io/rabbitmq/rabbitmq-server/ubuntu xenial InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY F6609E60DC62814E
W: Failed to fetch https://packagecloud.io/rabbitmq/rabbitmq-server/ubuntu/dists/xenial/InRelease The following signatures couldn't be verified because the public key is not available: NO_PUBKEY F6609E60DC62814E
W: Some index files failed to download. They have been ignored, or old ones used instead.
In addition to updating the 69464050 key, I also had to add the rabbitmq repo key
# inside the Vagrant VM
$ curl -L https://packagecloud.io/rabbitmq/rabbitmq-server/gpgkey | sudo apt-key add -
(taken from https://packagecloud.io/rabbitmq/rabbitmq-server/install#manual-deb)
I also had trouble around NFS in Linux where mounting the NFS shared folders would time out. I had to allow the ports used by nfs-server and other supporting services in my firewall.
On Arch, I edited my /etc/nfs.conf to the following so the ports for lockd, mountd, statd will remain fixed (they seem to be random by default)
[lockd]
port=35107
udp-port=42011
[mountd]
port=20048
[nfsd]
udp=y
vers3=y
[statd]
port=55327
I use ufw so I next added a custom app entry by creating a file /etc/ufw/applications.d/ufw-custom (adding a custom ufw app wasn’t necessary but I wanted to keep the ports grouped and labeled):
[NFS-others]
title=NFS supporting services
description=mountd, statd and lockd. Needed by NFS.
ports=35107,20048,55327/tcp|42011,20048,55327/udp
Then enabled both ufw apps NFS (came as default) and NFS-others.
$ sudo ufw allow NFS NFS-others
1 Like
I got an error on provisioning:
default: fatal: [localhost]: FAILED! => {"changed": true, "cmd": "gem install rubygems-update && update_rubygems", "delta": "0:00:01.855971", "end": "2018-12-20 12:20:24.791334", "failed": true, "rc": 1, "start": "2018-12-20 12:20:22.935363", "stderr": "ERROR: Error installing rubygems-update:\n\trubygems-update requires Ruby version >= 2.3.0.", "stderr_lines": ["ERROR: Error installing rubygems-update:", "\trubygems-update requires Ruby version >= 2.3.0."], "stdout": "", "stdout_lines": []}
Updating the ansible task to gem install "rubygems-update:< 3.0" helps.
I had to test something in my docker devstack against hawthorn, but it took too long to manually roll back migrations for every app. I wrote this script to attempt rolling back all migrations. Of course some can fail, but this helps speed up the proccess.
#!/bin/bash
shopt -s globstar
MIGRATIONS=$(git diff --name-only open-release/hawthorn.1 master **/migrations)
APPNAMES=$(
for MIGRATION in $MIGRATIONS
do
basename $(dirname $(dirname $MIGRATION))
done|sort|uniq
)
for APP in $APPNAMES
do
LAST_MIGRATION=$(git ls-tree -r open-release/hawthorn.1 --name-only|egrep $APP/migrations/'[0-9]+_'|tail -n1)
LAST_MIGRATION=$(basename $LAST_MIGRATION)
/edx/bin/{python,manage}.ecommerce migrate $APP ${LAST_MIGRATION//.py}
done
1 Like