With this change, containers are no longer run as "root" but as unprivileged
users. This is necessary in some environments, notably some Kubernetes
clusters.
To make this possible, we need to manually fix bind-mounted volumes in
docker-compose. This is pretty much equivalent to the behaviour in Kubernetes,
where permissions are fixed at runtime if the volume owner is incorrect. Thus,
we have a consistent behaviour between docker-compose and Kubernetes.
We achieve this by bind-mounting some repos inside "*-permissions" services.
These services run as root user on docker-compose and will fix the required
permissions, as per build/permissions/setowner.sh These services simply do not
run on Kubernetes, where we don't rely on bind-mounted volumes. There, we make
use of Kubernete's built-in volume ownership feature.
With this change, we get rid of the "openedx-dev" Docker image, in the sense
that it no longer has its own Dockerfile. Instead, the dev image is now simply
a different target in the multi-layer openedx Docker image. This makes it much
faster to build the openedx-dev image.
Because we declare the APP_USER_ID in the dev/docker-compose.yml file, we need
to pass the user ID from the host there. The only way to achieve that is with a
tutor config variable. The downside of this approach is that the
dev/docker-compose.yml file is no longer portable from one machine to the next.
We consider that this is not such a big issue, as it affects the development
environment only.
We take this opportunity to replace the base image of the "forum" image. There
is now no need to re-install ruby inside the image. The total image size is
only decreased by 10%, but re-building the image is faster.
In order to run the smtp service as non-root, we switch from namshi/smtp to
devture/exim-relay. This change should be backward-compatible.
Note that the nginx container remains privileged. We could switch to
nginxinc/nginx-unprivileged, but it's probably not worth the effort, as we are
considering to get rid of the nginx container altogether.
Close #323.
There is too much information in each of the local/k8s/dev docs pages. The
"guides" that are listed in each one of those pages are moved either to "common
tasks" or to a dedicated "tutorials" section. This paves the way for more
comprehensive tutorials, where we describe how to run the latest master
branches of Open edX.
I am well aware that, as they stand, the tutorials are of poor quality and
should be rewritten. This is a task for another day/commit. For now, we only
move the contents to a separate part of the docs.
Also, we should add a "reference" section to the docs, where we add the result
of `tutor <subcommand> --help`.
It should be unnecessary to build a custom openedx-dev Docker image. All tests
can run from within the dev Docker image, with a couple additional environment
variables.
We remove security patches and custom fixes which are now part of koa.3.
We take the opportunity to make it possible to build the openedx Docker image
without relying on a corresponding openedx-i18n repo tag: often, we want to
test whether the image simply builds successfully, and we don't need up-to-date
translations. For those cases, it's now possible to pass the `-a
OPENEDX_I18N_VERSION=oldertag` build argument.
First, allow using custom Django settings on a development
environment (as documented but not implemented), setting it to the
correct value of `tutor.development`. Prior to this, `tutor dev
runserver lms` would default to `tutor.production` when on a custom edX
branch.
Second, fix the documentation so the correct environment variable is
described, at the same time removing an option that doesn't seem to work.
See discussion: https://discuss.overhang.io/t/koa-dev-lms-doesnt-find-static-content/1250
We manage to get unit tests to run in a dedicated openedx-test container. Only
35 tests are failing (out of 17k). I suspect these tests are also failing in
the devstack.
This introduces a new dev/local command:
tutor dev bindmount CONTAINER PATH
And a new volume syntax:
tutor dev run --volume=PATH CONTAINER
This syntax automatically bind-mounts folders from the tutorroot/volumes
directory, which is pretty nifty.
- 💥[Improvement] Upgrade Open edX to Koa
- 💥 Setting changes:
- The ``ACTIVATE_HTTPS`` setting was renamed to ``ENABLE_HTTPS``.
- Other ``ACTIVATE_*`` variables were all renamed to ``RUN_*``.
- The ``WEB_PROXY`` setting was removed and ``RUN_CADDY`` was added.
- The ``NGINX_HTTPS_PORT`` setting is deprecated.
- Architectural changes:
- Use Caddy as a web proxy for automated SSL/TLS certificate generation:
- Nginx no longer listens to port 443 for https traffic
- The Caddy configuration file comes with a new ``caddyfile`` patch for much simpler SSL/TLS management.
- Configuration files for web proxies are no longer provided.
- Kubernetes deployment no longer requires setting up a custom Ingress resource or custom manager.
- Gunicorn and Whitenoise are replaced by uwsgi: this increases boostrap performance and makes it no longer necessary to mount media folders in the Nginx container.
- Replace memcached and rabbitmq by redis.
- Additional features:
- Make it possible to disable all plugins at once with ``plugins disable all``.
- Add ``tutor k8s wait`` command to wait for a pod to become ready
- Faster, more reliable static assets with local memory caching
- Deprecation: proxy files for Apache and Nginx are no longer provided out of the box.
- Removed plugin `{{ patch (...) }}` statements:
- "https-create", "k8s-ingress-rules", "k8s-ingress-tls-hosts": these are no longer necessary. Instead, declare your app in the "caddyfile" patch.
- "local-docker-compose-nginx-volumes": this patch was primarily used to serve media assets. The recommended is now to serve assets with uwsgi.
When I tried running `openedx-assets build` on my `tutor dev lms` machine, I got an error:
```
openedx@1dfe0ece7805:~/edx-platform$ openedx-assets build --env=dev
mkdir_p path('common/static/common/js/vendor')
mkdir_p path('common/static/common/css')
mkdir_p path('common/static/common/css/vendor')
Copying vendor files into static directory
Traceback (most recent call last):
File "/openedx/bin/openedx-assets", line 218, in <module>
main()
File "/openedx/bin/openedx-assets", line 89, in main
args.func(args)
File "/openedx/bin/openedx-assets", line 94, in run_build
run_npm(args)
File "/openedx/bin/openedx-assets", line 117, in run_npm
assets.process_npm_assets()
File "/openedx/edx-platform/pavelib/assets.py", line 643, in process_npm_assets
copy_vendor_library(library)
File "/openedx/edx-platform/pavelib/assets.py", line 614, in copy_vendor_library
raise Exception(u'Missing vendor file {library_path}'.format(library_path=library_path))
Exception: Missing vendor file node_modules/backbone.paginator/lib/backbone.paginator.js
```
As suggested in [this topic](https://discuss.overhang.io/t/issue-with-paver-update-assets/641) I had to run `npm install` to get the packages it tries to copy from. That makes sense, so I think it should be part of the instructions here.
Here, we upgrade the Open edX platform from Ironwood to Juniper. This
upgrade does not come with many feature changes, but there are many
technical improvements under the hood:
- Upgrade from Python 2.7 to 3.5
- Upgrade from Mongodb v3.2 to v3.6
- Upgrade Ruby to 2.5.7
We took the opportunity to completely rething the way locally running
platforms should be accessed for testing purposes. It is no longer
possible to access a running platform from http://localhost and
http://studio.localhost. Instead, users should access
http://local.overhang.io and https://studio.local.overhang.io. This
drastically simplifies internal communication between Docker containers.
To upgrade, users should simply run:
tutor local quickstart
For Kubernetes platform, the upgrade process is outlined when running:
tutor k8s upgrade --from=ironwood
There are too many patches on top of ironwood.2, and it's not practical
to pull them all one by one. We still want to build on top of a specific
version, and not a branch, so we use a dirty hack to guarantee that the
docker image is properly rebuilt by CI when we change it.
By de-duplicating the code between dev.py and local.py, we are able to
support more docker-compose run/up/stop options passed from tutor. To do
so, we had to disable some features, such as automatically mounting the
edx-platform repo when the TUTOR_EDX_PLATFORM_PATH environment variable
was defined.
The `dev` commands now rely on a different openedx-dev docker image.
This gives us multiple improvements:
- no more chown in base image
- faster chown in development
- mounted requirements volume in development
- fix static assets issues
- bundled ipdb/vim/... packages, which are convenient for development
Close #235
- More concise table of contents
- New intro
- Simpler make commands
- Fix a couple typos here and there
- Get rid of the default github issue template, and start using the
template created online.
Now that the correct webpack settings are loaded by the `update_assets`
command in Ironwood, we can stop relying on the `openedx-assets` script.
Actually, we could probably remove it.
USERID environment variable was no longer passed to docker image in
development mode.
We take the opportunity to improve the documentation regarding the dev
environment.
Close #177.
Replace all make commands by a single "tutor" binary. Environment and
data are all moved to ~/.tutor/local/share/tutor. We take the
opportunity to add a web UI and revamp the documentation.
This is a complete rewrite.
Close #121.
Close #147.
