Running `tutor config save` with an outdated version of pycryptodome was
failing with the following error:
Error: Missing configuration value: 'Crypto.PublicKey.RSA.RsaKey object' has no attribute 'dq'
This is because the "dq" attribute was only introduced in pycryptodome
3.17.0: https://www.pycryptodome.org/src/changelog#january-2023
To resolve this issue we bump the minimum requirements.
Close #962
New hook Action that allows tutor plugins to interact with the configuration at the time of the interactive questionnaire that happens during `tutor local/dev launch`.
These changes make to possible to run:
tutor mounts add /path/to/my-xblock
The xblock directory with then be auto-magically bind-mounted in the
"openedx" image at build time, and the lms*/cms* containers at run time.
This makes it effectively possible to work as a developer on
edx-platform requirements.
We take the opportunity to move some openedx-specific code to a
dedicated module.
Close https://github.com/openedx/wg-developer-experience/issues/177
The new domain name points to 127.0.0.1, just like the previous one. We
keep the local.overhang.io domain names for backward compatibility. In
the future, we hope to migrate to "*.openedx.io" but that will not
happen before Redwood.
Close #945
Bumping the `OPENEDX_COMMON_VERSION` in the master branch usually
creates a conflict when we merge the change in the nightly branch. To
avoid this conflict, we add some logic to the `OPENEDX_COMMON_VERSION`.
This change should be invisible for most users.
This partially addresses issue #936.
MySQL 8 defaults to a binlog expiry period of 2592000 seconds
(30 days), which for Tutor/Open edX purposes can be considered
excessive.
On the one hand, it is unlikely that a MySQL server configured for
Tutor uses MySQL replication at all (considering that up until Tutor
15 and MySQL 5.7, the binlog was disabled by default, rendering
replication impossible). Even if it does, a replica lagging more than
two days behind the primary server would be unacceptable.
Likewise, it is unlikely that an Open edX database is backed up less
than once a day, thus is is unlikely that Open edX admins would
benefit from the ability to do point-in-time restore over a 30-day
period.
On the other hand, having a 30-day binlog expiry period can
considerably increase the storage space requirements for the MySQL
container, particularly on busy Open edX platforms. When left
unchecked, this can even cause the MySQL container to run into "No
space left on device" situations, disabling the MySQL database
altogether. Thus, the MySQL default settings are likely to be a net
disadvantage for Open edX admins.
Finally, all of the above considerations apply only if the Open edX
administrator has chosen to run their own MySQL and not opted for a
DBaaS solution like AWS RDS.
Thus, it should be acceptable to run with a reduced binlog expiry
period of 3 days (rather than 30) by default.
Therefore, inject the --binlog-expire-logs-seconds=259200 argument
into the Tutor-generated command to start mysqld.
Reference:
https://dev.mysql.com/doc/refman/8.0/en/replication-options-binary-log.html#sysvar_binlog_expire_logs_seconds
On macOS, building the "openedx-dev" Docker image resulted in an image that required more than 600 GB of disk space. This was due to the `adduser` command which was called with a user ID of 2x10⁹ (on macOS only). This resulted in a very large /var/log/faillog file, hence the image size.
Related upstream discussion: https://github.com/moby/moby/issues/5419
Close https://github.com/openedx/wg-developer-experience/issues/178
On Oct. 10, py2neo package was abruptly removed from pypi, GitHub, and
the py2neo website now displays just a super funny meme: https://py2neo.org/
Yes, we should get rid of that dependency, but we are still supposed to
support existing users. So we install py2neo from our fork.
The LMS was overriding CMS_BASE properly, but Studio (CMS) configuration
was not. That meant that Studio's CMS_BASE in dev mode was using the
devstack default of localhost:18010 (because this is what's defined in
edx-platform). This in turn broke parts of Studio that use this value,
such as the XBlock v2 API (/api/xblock/v2).
This commit derives the value of the CMS_BASE Django setting from
Tutor's CMS_HOST config value, in the same way that the LMS does it.
I added `rsync` to Transfer the configuration, environment, and platform data from server 1 to server 2 command so that we can we able to transfer data.
I found that `-avr` options fit well to it.
We upgrade nodeenv as an attempt to fix incomplete reads.
From time to time we face the following error:
#67 [linux/amd64 nodejs-requirements 2/4] RUN nodeenv /openedx/nodeenv --node=16.14.0 --prebuilt
#67 0.338 * Install prebuilt node (16.14.0) .Incomplete read while readingfrom https://nodejs.org/download/release/v16.14.0/node-v16.14.0-linux-x64.tar.gz#67 204.1 .
#67 204.1 Traceback (most recent call last):
#67 204.1 File "/openedx/venv/bin/nodeenv", line 8, in <module>
#67 204.1 sys.exit(main())
#67 204.1 File "/openedx/venv/lib/python3.8/site-packages/nodeenv.py", line 1104, in main
#67 204.1 create_environment(env_dir, args)
#67 204.1 File "/openedx/venv/lib/python3.8/site-packages/nodeenv.py", line 980, in create_environment
#67 204.1 install_node(env_dir, src_dir, args)
#67 204.1 File "/openedx/venv/lib/python3.8/site-packages/nodeenv.py", line 739, in install_node
#67 204.1 install_node_wrapped(env_dir, src_dir, args)
#67 204.1 File "/openedx/venv/lib/python3.8/site-packages/nodeenv.py", line 762, in install_node_wrapped
#67 204.1 download_node_src(node_url, src_dir, args)
#67 204.1 File "/openedx/venv/lib/python3.8/site-packages/nodeenv.py", line 602, in download_node_src
#67 204.1 with ctx as archive:
#67 204.1 File "/opt/pyenv/versions/3.8.15/lib/python3.8/contextlib.py", line 113, in __enter__
#67 204.1 return next(self.gen)
#67 204.1 File "/openedx/venv/lib/python3.8/site-packages/nodeenv.py", line 573, in tarfile_open
#67 204.1 tf = tarfile.open(*args, **kwargs)
#67 204.1 File "/opt/pyenv/versions/3.8.15/lib/python3.8/tarfile.py", line 1601, in open
#67 204.1 saved_pos = fileobj.tell()
#67 204.1 AttributeError: 'bytes' object has no attribute 'tell'
This change was added to 1.8.0 as an attempt to resolve the issue: https://github.com/ekalinin/nodeenv/pull/329
We are not sure it will work every time, but it can't hurt.
Pound keys were interpreted as comments. This is annoying when we want
to parse html color codes, such as in:
$ tutor config save --set "INDIGO_PRIMARY_COLOR=#225522"
$ tutor config printvalue INDIGO_PRIMARY_COLOR
null
Close #866.
Now that sphinx_rtd support docutils>=0.19 we can drop that max version
requirement. But we need to limit sphinx max version because they
removed python 3.8 support before EOL.
This is a follow-up fix to #819, where the corresponding change was
added to the mysqld invocation in the "tutor local" (that is,
docker-compose) deployment method, but omitted from its "tutor k8s"
equivalent.
This fix is for a rather serious issue that affects users who upgrade
from Olive to Palm. The client mysql charset and collation was
incorrectly set to utf8mb4, while the server stil runs utf8mb3. Only
users who run the mysql container are affected.
To resolve this issue, we explicitely configure the client to use the
utf8mb3 charset/collation.
Important note: users who have somehow managed to upgrade from olive to
Palm before may find themselves in an undefined state. They might have
to fix their mysql data manually. Same thing for users who launched Palm
from scratch; although, according to my preliinary tests, they should be
able to downgrade their connection from utf8mb4 to utf8mb3 without
issue.
In addition, we upgrade to mysql 8.1.0. Among many other fixes, this
avoids a server restart after the upgrade:
> An in-place upgrade from MySQL 5.7 to MySQL 8.0, without a server
> restart, could result in unexpected errors when executing queries on
> tables. This fix eliminates the need to restart the server between the
> upgrade and queries. (Bug #35410528)
https://dev.mysql.com/doc/relnotes/mysql/8.0/en/news-8-0-34.html
See also the 8.1.0 release notes:
https://dev.mysql.com/doc/relnotes/mysql/8.1/en/news-8-1-0.html
Close #887.
`tutor ... do settheme default` is meant to revert to the
default theme. However, in its current implementation, it
creates SiteTheme objects pointing to a theme named "default",
which doesn't exist, resulting in errors like:
Theme dirs:
[Path('/openedx/themes')]]
Traceback (most recent call last):
File "/openedx/edx-platform/openedx/core/djangoapps/theming/helpers.py", line 204, in get_current_theme
themes_base_dir=get_theme_base_dir(site_theme.theme_dir_name),
File "/openedx/edx-platform/openedx/core/djangoapps/theming/helpers.py", line 242, in get_theme_base_dir
raise ValueError(
ValueError: Theme 'default' not found in any of the following themes dirs,
This works from the perspective of the user, because a missing theme is
treated as the default theme. However, the errors are unneccesary &
confusing.
By simply deleting & not recreating SiteTheme objects instead,
we are able to revert to the default theme while keeping the
logs clear of theming errors.
Manual configuration via the `MOUNTS` setting was inconvenient. We
(re)introduce a new(ish) `tutor mounts` command. Old timers will perhaps
remember that we used to have a `tutor bindmount` command. Well, it's
back! But better and different.
Templated hooks we almost completely useless, so we get rid of them.
This allows us to get rid entirely of hook names and hook indexes, which
makes the whole implementation much simpler. Hook removal (with
`clear_all`) is achieved thanks to weak references.
This is an important change, where we get remove the previous `--mount`
option, and instead opt for persistent bind-mounts.
Persistent bind mounts have several advantages:
- They make it easier to remember which folders need to be bind-mounted.
- Code is *much* less clunky, as we no longer need to generate temporary
docker-compose files.
- They allow us to bind-mount host directories *at build time* using the
buildx `--build-context` option.
- The transition from development to production becomes much easier, as
images will automatically be built using the host repo.
The only drawback is that persistent bind-mounts are slightly less
portable: when a config.yml file is moved to a different folder, many
things will break if the repo is not checked out in the same path.
For instance, this is how to start working on a local fork of
edx-platform:
tutor config save --append MOUNTS=/path/to/edx-platform
And that's all there is to it. No, this fork will be used whenever we
run:
tutor images build openedx
tutor local start
tutor dev start
This change is made possible by huge improvements in the build time
performance. These improvements make it convenient to re-build Docker
images often.
Related issues:
https://github.com/openedx/wg-developer-experience/issues/71https://github.com/openedx/wg-developer-experience/issues/66https://github.com/openedx/wg-developer-experience/issues/166
It was useless to create a *-permissions job for every application.
Instead, we create a single "permissions" service. It can be extended
via the "docker-compose-permissions-command" patch.
Instead, the compose plugin must be installed.
We deprecate docker-compose because v1 is not supported starting from
the end of June 2023.
See "evolution of compose": https://docs.docker.com/compose/compose-v2/
Among other changes: ORA2 file uploads were stored in a folder named
"SET-ME-PLEASE (ex. bucket-name)" (sigh). With this change, the folder
should be automatically renamed to "openedxuploads". This issue has been
occuring since June 2019... (sigh²)
Close #707
See:
https://github.com/overhangio/tutor/actions/runs/5260213022/jobs/9506811909#step:9:33
`scriv github-release --repo=overhangio/tutor` causes the following
error:
Traceback (most recent call last):
File "/opt/hostedtoolcache/Python/3.7.16/x64/bin/scriv", line 8, in
<module>
sys.exit(cli())
File
"/opt/hostedtoolcache/Python/3.7.16/x64/lib/python3.7/site-packages/click/core.py",
line 1130, in __call__
return self.main(*args, **kwargs)
File
"/opt/hostedtoolcache/Python/3.7.16/x64/lib/python3.7/site-packages/click/core.py",
line 1055, in main
rv = self.invoke(ctx)
File
"/opt/hostedtoolcache/Python/3.7.16/x64/lib/python3.7/site-packages/click/core.py",
line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File
"/opt/hostedtoolcache/Python/3.7.16/x64/lib/python3.7/site-packages/click/core.py",
line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File
"/opt/hostedtoolcache/Python/3.7.16/x64/lib/python3.7/site-packages/click/core.py",
line 760, in invoke
return __callback(*args, **kwargs)
File
"/opt/hostedtoolcache/Python/3.7.16/x64/lib/python3.7/site-packages/scriv/ghrel.py",
line 99, in github_release
config=scriv.config,
File
"/opt/hostedtoolcache/Python/3.7.16/x64/lib/python3.7/site-packages/jinja2/environment.py",
line 1301, in render
self.environment.handle_exception()
File
"/opt/hostedtoolcache/Python/3.7.16/x64/lib/python3.7/site-packages/jinja2/environment.py",
line 936, in handle_exception
raise rewrite_traceback_stack(source=source)
File "<template>", line 3, in top-level template code
TypeError: 'Version' object is not subscriptable
in some cases, tutor might run inside a pod, which that pod has access to a cluster via role binding and a service account. this way, there's no ./kube/config file, but kubectl commands run with no issue.
Close #843
in some cases, tutor might run inside a pod, which that pod has access to a cluster via role binding and a service account. this way, there's no ./kube/config file, but kubectl commands run with no issue.
Close #843
Installing from source triggers a warning on pip 23.0.1 if
pyproject.toml is not present. Building does not require any special
dependencies, so we just add a simple pyproject.toml file.
Close #836
This is useful when the generation of a macOS binary has failed, for
instance.
In addition, we restore checking for the `gh` utility. This is necessary
when running CI locally with `act`.
In addition, we cache pip dependencies on github release and test runs.
This paves the way for `docker buildx build` and better caching.
For instance, with this change you can try out the following plugin,
which should make image building much faster in CI:
https://gist.github.com/regisb/4049622ec4b48cbd48c89ec708dc5252
(not ready for production just yet, we still need to build and push the
images)
In the output of `plugins list` it's difficult to see which plugins are
enabled at a glance. This change adds a more visible checkmark to the
output.
Ex:
$ tutor plugins list
NAME STATUS VERSION
cairn ✅ enabled 15.0.3
discovery installed 15.0.0
ecommerce installed 15.0.1
forum installed 14.0.0
mfe ✅ enabled 15.0.5
minio installed 15.1.0
Before this commit, setting up an edx-platform development environment
took multiple steps:
tutor dev launch
tutor dev run --mount=/path/to/edx-platform lms bash
>> pip install -e .
>> npm clean-install
>> openedx-assets build --env=dev
This commit moves the steps under ``run`` into an init task, which
is automatically run by ``launch``. Thus, setup is now one command:
tutor dev launch --mount=edx-platform
These extra init steps are only applicable when bind-mounting
edx-platform (because bind-mounting the repository overrides
some important artifacts that exist on the image, which must be
re-generated). Thus, the new init tasks exists early if it detects
that it is *not* operating on a bind-mounted repository.
Finally, we try to simplify the Open edX development docs so that
it is clearer how bind-mounting fits into the development process.
These bind-mounts:
* ../build/openedx/themes:/openedx/themes
* ../build/openedx/requirements:/openedx/requirements
existed in the dev lms and cms containers, but they did
not exist in the lms-job and cms-job containers.
This means that themes and requirements that were *built into the
image* would exist in the job containers, but live updates to the
themes and requirements would not apply.
To resolve this, we set ``volumes:`` on the lms-job and cms-job
services so that they match the volumes for the normal lms and
cms services.
Part of: https://github.com/openedx/wg-developer-experience/issues/146
Closes: https://github.com/openedx/wg-developer-experience/issues/152
This works around (but does not close) these related issues:
* https://github.com/openedx/wg-developer-experience/issues/150
* https://github.com/openedx/wg-developer-experience/issues/151
The lms-job and cms-job services were configured to use
{{ DOCKER_IMAGE_OPENEDX }} rather than {{ DOCKER_IMAGE_OPENEDX_DEV }}.
This means that when running jobs in dev mode, a la:
tutor dev do init
a production image would be used, to the user's surprise.
On some OS, high ulimit causes some container to use a lot of memory. We
do not attempt to resolve this issue in Tutor because this is a
mysql/uwsgi issue. Instead, we explain how to resolve it in the
troubleshooting docs.
Close #671.
When a job is invoked, we now replace the job in k8s/jobs.yml
instead of rewriting jobs.yml to only contain the relevant
job. This allows patchStrategicMerge to work for jobs.
We implement this TEP: https://discuss.openedx.org/t/tutor-enhancement-proposal-tep-plugin-indices/8182
With plugin indexes, tutor users can install and upgrade plugins directly from indexes:
tutor plugins install ecommerce
tutor plugins index add contrib
tutor plugins install codejail
tutor plugins upgrade all
This change has been long in the coming \o/
Unfortunately, previous reqs upgrade was not compatible with python 3.7
because isort dropped support for that "almost EOL" version:
https://github.com/PyCQA/isort/pull/2064
Users want to be able to override the request `max_size` to upload
larger files. But they will not be able to if the patch is placed after
the `request` directive. So we move the patch statement before the
directive. Also, we wrap the `request_body` directives within `handle`
statements. If not, then different sizes are not managed properly.
To override the max upload size in the cms, add the following to the
"caddyfile-cms" patch:
handle_path /import/* {
request_body {
max_size 500MB
}
}
See discussion:
https://discuss.openedx.org/t/how-to-update-caddyfile-using-tutor-plugin/8944
The hooks API had several issues which are summarized in this comment:
https://github.com/openedx/wg-developer-experience/issues/125#issuecomment-1313553526
1. "consts" was a bad name
2. "hooks.filters" and "hooks.Filters" could easily be confused
3. docs made it difficult to understand that plugin developers should use the catalog
To address these issues, we:
1. move "consts.py" to "catalog.py"
2. Remove "hooks.actions", "hooks.filters", "hooks.contexts" from the API.
3. re-organize the docs and give better usage examples in the catalog.
This change is a partial fix for https://github.com/openedx/wg-developer-experience/issues/125
Adds `from __future__ import annotations` to the top of every module,
right below the module's docstring. Replaces any usages of t.List,
t.Dict, t.Set, t.Tuple, and t.Type with their built-in equivalents:
list, dict, set, tuple, and type. Ensures that make test still passes
under Python 3.7, 3.8 and 3.9.
The LMS and CMS were producing lots of logs similar to:
cms_1 | 2023-01-17 15:30:11,359 INFO 7 [openedx.core.djangoapps.cors_csrf.helpers] [user 7] [ip 31.223.46.44] helpers.py:64 - Origin 'https://studio.demo.openedx.overhang.io' was not in `CORS_ORIGIN_WHITELIST`; full referer was 'https://studio.demo.openedx.overhang.io/learning/course/course-v1:edX+DemoX+Demo_Course/home' and requested host was 'studio.demo.openedx.overhang.io'; CORS_ORIGIN_ALLOW_ALL=False
These warnings are produced by openedx.core.djangoapps.cors_csrf.helpers. I
don't think they indicate any problem, but they pollute the logs. They are
resolved by adding the "http(s)://<lms/cms host>" to CORS_ORIGIN_WHITELIST in
production, so we did just that.
When a user registers, they receive a confirmation email. This email contained
two links to "https://example.com/..." urls. This was caused by the fact that
the default site, indicated by SITE_ID=1, was example.com. We resolve this
issue by setting instead SITE_ID=2, which should point to the site with the LMS
domain name.
This is a potentially breaking change for platforms that have manually set to 1
the id of the LMS site in the database. These platforms should now set
SITE_ID=1 via a plugin.
Alternatives we have considered include modifying the id field of the LMS site
in the database. Unfortunately such a change would have important consequences,
as the site ID is used as a foreign key for other models.
Note that non-https sites still include https links in the registration emails.
This is because the "https" scheme is hardcoded by the "ensure_url_is_absolute"
utility function. So there is nothing we can do about this without making
changes upstream.
Close #572.
Adds `from __future__ import annotations` to the top of every module,
right below the module's docstring. Replaces any usages of t.List,
t.Dict, t.Set, t.Tuple, and t.Type with their built-in equivalents:
list, dict, set, tuple, and type. Ensures that make test still passes
under Python 3.7, 3.8 and 3.9.
Tutor binary releases were no longer compatible with Ubuntu 20.04 since the
ubuntu-latest image was 22.04 on GitHub.
The error was:
[7893] Error loading Python lib '/tmp/_MEIcyvkMV/libpython3.7m.so.1.0': dlopen: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.35' not found (required by /tmp/_MEIcyvkMV/libpython3.7m.so.1.0)
We fix this issue by downgrading the GitHub image with which we make the
release.
Close #765.
This removes an openedx/edx-platform commit backported as a patch to tutor to olive.1 release
Since the commit is already merged into edx-platform:master branch used
by tutor nightly, there is no further need for it.
- [Improvement] Upgrade ipdb and ipython packages in the openedx development image. (by @regisb)
- [Improvement] Skip unnecessary image building in development. This should make `tutor dev launch` slightly faster. (by @regisb)
- [Bugfix] Fix Authn MFE login in development by disabling enterprise integration. (by @regisb)
- [Bugfix] Fix "Invalid value for ‘--from’" when running `tutor local upgrade --from=nutmeg`. If you are facing this error, just run `tutor local launch` and your platform should be automatically upgraded.
- [Bugfix] Fix "TypeError: Parameters to Generic[...] must all be type variables" error. This error may occur when upgrading from a very old installation of Tutor. It is due to an old version of the typing-extensions package.
- 💥[Deprecation] Get rid of the `quickstart` command. v15.0.0 introduced a deprecation warning, but we actually want users to stop using this command. Instead, use `launch` (by @regisb).
- [Improvement] Backfill persistent grades during upgrade from Nutmeg. If you observe missing grades after the upgrade from Nutmeg, run `tutor local upgrade --from=nutmeg`. (by @regisb)
In development, login via the authn mfe was broken because of explicit
enterprise integration:
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=18000): Max retries exceeded with url: /enterprise/api/v1/enterprise-learner/?username=regis (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fd4c02b8a90>: Failed to establish a new connection: [Errno 111] Connection refused'))
See: https://discuss.overhang.io/t/tutor-login-fail-in-new-version/3083
I have no idea why we decided to kickstart a separate build. The image will be
built anyway at the next step because we run `docker compose up --build` in
`tutor dev start`.
The build step was introduced in this PR: https://github.com/overhangio/tutor/pull/627
v15.0.0 changelog includes entries from v14.0.0 (!!!). I assume this is because
we did not delete these entries from the nightly changelog during the v14
upgrade.
Close #761.
People running typing-extensions==3.10 faced this error for just any tutor command:
$ tutor version
...
Traceback (most recent call last):
File "/usr/local/bin/tutor", line 5, in <module>
from tutor.commands.cli import main
File "/usr/local/lib/python3.8/dist-packages/tutor/commands/cli.py", line 7, in <module>
from tutor import exceptions, fmt, hooks, utils
File "/usr/local/lib/python3.8/dist-packages/tutor/hooks/__init__.py", line 7, in <module>
from . import actions, contexts, filters, priorities
File "/usr/local/lib/python3.8/dist-packages/tutor/hooks/actions.py", line 18, in <module>
class ActionCallback(Contextualized, t.Generic[P]):
File "/usr/lib/python3.8/typing.py", line 261, in inner
return func(*args, **kwds)
File "/usr/lib/python3.8/typing.py", line 890, in __class_getitem__
raise TypeError(
TypeError: Parameters to Generic[...] must all be type variables
We fix this error by requiring a more recent version of typing-extensions.
See: https://discuss.openedx.org/t/tutor-v15-python-error-when-running-on-quickstart/8910/2
GitHub Actions sometimes runs `make tests` as root; e.g: in the release script.
There were unit tests that were breaking in that scenario. I have no idea why
tests were not breaking in the test.yml workflow.
- [Bugfix] Fix `jinja2.exceptions.TemplateSyntaxError: expected token 'end of statement block', got '|'` error by bumping the minimum required version of the Jinja2 package.
- [Feature] Add support for MongoDB SSL, authentication source, mechanism and replica set via the `MONGODB_USE_SSL`, `MONGODB_AUTH_MECHANISM`, `MONGODB_AUTH_SOURCE`, `MONGODB_REPLICA_SET` settings. (by @zakum1 and @regisb)
- [Bugfix] Fix tag of "openedx" development Docker image. Previously, this Docker tag did not include the Tutor version. As a consequence, a different cached image could be used in some cases. For instance: when running `tutor dev run` commands. Now, the image tag is "openedx-dev:TUTOR_VERSION".
- [Bugfix] Fix name of Swahili locale: it is "sw-ke" and not "sw" (by @regisb).
- [Security] Apply drag-n-drop v2 xblock [security patch](https://discuss.openedx.org/t/upcoming-security-release-xblock-drag-and-drop-v2/8768/7). (by @regisb)
When running:
tutor dev run -m /path/to/edx-platform lms
pip install -r requirements/edx/development.txt
I realised that I was re-installing packages that should already have been
present in the image. The reason for that was that I was running an outdated
version of the dev version of the openedx Docker image. This happens because
`tutor dev run` does not trigger an image re-build.
We solve this issue by pinning the openedx dev Docker image tag to the current
tutor version.
This change builds upon a previously proposed PR:
https://github.com/overhangio/tutor/pull/437
There was another long conversation about this topic here:
https://github.com/overhangio/tutor-forum/pull/10#issuecomment-1314799915
We could have supported the MongoDB auth/replica set/ssl parameters as part of
the MongoDB host URI, but then this URI is not supported in the forum plugin,
which uses an old version of the mongoid client. We were hoping that the client
would have been upgraded by now, but it's not been upgraded for a long time.
The changes introduced here are 100% backward-compatible. The forum plugin will
have to be updated to take into account the new parameters.
The following syntax is only supported in jinja2>=2.10:
{% set jwt_rsa_key | rsa_import_key %}{{ JWT_RSA_PRIVATE_KEY }}{% endset %}
Thus, we bump the minimal working version of jinja2 in the base requirements.
See discussion: https://discuss.openedx.org/t/error-while-tutor-local-quickstart/8796
- [Improvement] Auto-completion of `plugins` and `config` arguments: `plugins enable/disable NAME`, `plugins install PATH`, `config save --set KEY=VAL`, `config save --unset KEY`, `config printvalue KEY`. (by @regisb)
- [Bugfix] Fix minimum click version (>= 8.0.0) when installing tutor from pip.
- [Improvement] Enable CORS by default for both LMS and CMS by moving those settings to the `common_all` partial. (by @arbrandes)
Changelog management was starting to be a hassle:
- there were conflicts every time a PR was merged
- there were conflicts every time we merged the nightly branch in the new
release branch, or vice versa.
Now, all changelog entries are stored as separate files in changelog.d,
including nightly. Nightly entries will be collected for every major release.
The ENABLE_CORS_HEADERS feature flag is already true for the LMS.
Instead of duplicating it for Studio via yaml settings, make this a
common Django setting to both LMS and CMS and all their environments.
- [Improvement] Auto-complete implicit `local/dev --mount /path/to/...` options. (by @regisb)
- 💥[Feature] Strong typing of action and filter hooks: this allows us to detect incorrect calls to `actions.add` or `filters.add` early. Strong typing forces us to break the `do` and `apply` API by removing the `context` named argument. Developers should replace `do(context=...)` by `do_from_context(..., )` (and similar for `apply`).
When typing `tutor local run --mount /path/to/edx-pl<TAB>`, the mount option
should be auto-completed to the full edx-platform repo path. That is, if shell
completion is enabled:
https://docs.tutor.overhang.io/install.html#shell-autocompletion
Here, we make sure that the implicit form of the `--mount` argument is properly
auto-completed. We are unable to get completion to work in the explicit form,
because args that include colons do not even reach the `shell_completion`
method.
Now that the mypy bugs have been resolved, we are able to define more precisely
and cleanly the types of Actions and Filters.
Moreover, can now strongly type named actions and hooks (in consts.py). With
such a strong typing, we get early alerts of hooks called with incorrect
arguments, which is nothing short of awesome :)
This change breaks the hooks API by removing the `context=...` argument. The
reason for that is that we cannot insert arbitrary arguments between `P.args,
P.kwargs`: https://peps.python.org/pep-0612/#the-components-of-a-paramspec
> A function declared as def inner(a: A, b: B, *args: P.args, **kwargs:
> P.kwargs) -> R has type Callable[Concatenate[A, B, P], R]. Placing
> keyword-only parameters between the *args and **kwargs is forbidden.
Getting the documentation to build in nitpicky mode is quite difficult... We
need to add `nitpick_ignore` to the docs conf.py, otherwise sphinx complains
about many missing class references. This, despite upgrading almost all doc
requirements (except docutils).
We introduce a new filter to implement custom commands in arbitrary containers.
It becomes easy to write convenient ad-hoc commands that users will
then be able to run either on Kubernetes or locally using a documented CLI.
Pluggable jobs are declared as Click commands and are responsible for
parsing their own arguments. See the new CLI_DO_COMMANDS filter.
Close https://github.com/overhangio/2u-tutor-adoption/issues/75
Nothing revolutionary here, we just implement the same priority queue that
existed in actions. It will be necessary to trigger init tasks in the right
order.
- [Security] Fix xblock ajax handler vulnerability. (by @regisb)
- [Improvement] Use web proxy gzip encoding to improve bandwidth. We
observe a 75% size reduction on the LMS dashboard. (by @ghassanmas)
Compressing assests would lead to readuce transfer size.
As testing with frontend-app-learning/Olive, the network traffic
before was about ~4MB, after this it became ~1MB.
This change was suggested by Google Lighthouse[1], there are of
course more suggestion but this was one the easiest and one of most
impactful.
Also check orignal PR overhangio/tutor-mfe/pull/64 for more
info.
[1]: https://web.dev/uses-text-compression
- [Improvement] Upgrade Scorm XBlock to v14.0.0. (by @regisb)
- 💥[Improvement] The Richie plugin was transferred to the Openfun organization; thus, it is no longer officially supported and it is removed from the default set of plugins that ships with
`pip install tutor[full]` or the Tutor pre-compiled binary. Users are encouraged to uninstall the `tutor-richie` Python package and install the `tutor-contrib-richie` package instead.
- [Feature] Upgrade edx-platform i18n strings to nutmeg.2. (by @regisb)
The `compilejsi18n` command was failing during image building because the
Open-edX package was not installed properly. The reason for that was an earlier
change where we got rid of the `pip install -r requirements/edx/local.in`
command. Installing the Open-edX package was part of this requirement file.
The local.in requirements file no longer exists, but we still need to `pip
install -e .` the edx-platform repo. To run this command we need both the
edx-platform repo and the virtualenv.
The good news is that there are no more local requirements in the base.txt
requirements file. This means that we no longer have to COPY the edx-platform
repo in the requirements installation step. Thus, changes in edx-platform will
no longer trigger a rebuild of the pip requirements; this means that re-builds
will be much faster when making changes to edx-platform.
Note that plugins that implemented the
"openedx-dockerfile-post-python-requirements" patch and that needed access to
the edx-platform repo will no longer work. Instead, these plugins should
implement the "openedx-dockerfile-pre-assets" patch. This scenario should be
very rare, though.
Close #726
The local requirements files does not exist since local requirements were all
removed from the edx-platform repo. As a consequence, the nightly build was
broken.
`quickstart` is being renamed to `launch` and deprecated in favor of
using `launch`. The `quickstart` function temporarily aliases to
`launch`. Further mentions of `quickstart` have been changed to
reference `launch` instead.
We are indicating that this change is breaking 💥 to encourage people to
migrate their scripts right away!
Strings could not be pulled from transifex because the file names were
incorrect. This is now fixed and we are now able to pull the i18n strings from
the nutmeg.2 tag.
Soon, running:
pip install -r ./requirements/edx/base.txt
in edx-platform will no longer install the local
project (that is, `-e .`). To prepare for that change,
we add the line:
pip install -e .
to the Dockerfile. This is backwards-compatible.
More details:
https://openedx.atlassian.net/browse/BOM-2575?focusedCommentId=613181
The pymongo dependency for edx-platform was updated (3.10.1 to 3.12.3)
in https://github.com/openedx/edx-platform/pull/30569
This caused the following error when running the edx-platform database
migration split_modulestore_django.0002_data_migration as part of
`tutor dev quickstart`:
pymongo.errors.ServerSelectionTimeoutError: client is configured to
connect to a replica set named '' but this node belongs to a set named
'None', Timeout: 30s, Topology Description: <TopologyDescription id:
62bdbaf182687350acf1aeec, topology_type: Single, servers:
[<ServerDescription ('mongodb', 27017) server_type: Unknown, rtt:
None, error=ConfigurationError("client is configured to connect to a
replica set named '' but this node belongs to a set named 'None'")>]>
This commit explicitly sets replicaSet to None to indicate that it's a
standalone MongoDB instance. I also had to remove the CONTENTSTORE entry
from auth.yml because edx-platform's devstack.py assumes it has a
non-null value (set in common.py), and devstack.py executes before
tutor's development.py can set this replicaSet value.
- [Feature] Add the `-m/--mount` option to `tutor dev quickstart`.
- [Bugfix] Fix `tutor dev start -m /path/to/frontend-app-learning` by introducing dev-specific `COMPOSE_DEV_TMP` and `COMPOSE_DEV_JOBS_TMP` filters (by @regisb).
- [Bugfix] Log the shell commands that Tutor executes more accurately. (by @kdmccormick)
- [Bugfix] `tutor dev quickstart` would fail under certain versions of docker-compose due to a bug in the logic that handled volume mounting. (by @kdmccormick)
- [Bugfix] The `tutor k8s start` command will succeed even when `k8s-override` and `kustomization-patches-strategic-merge` are not specified. (by @edazzocaisser)
- [BugFix] `kubectl wait` checks deployments instead of pods as it could hang indefinitely if there are extra pods in a broken state. (by @keithgg)
The -m/--mount option makes it possible to bind-mount volumes at runtime. The
volumes are declared in a local/docker-compose.tmp.yml file. The problem with
this approach is when we want to bind-mount a volume to a service which is
specific to the dev context. For instance: the "learning" service when the MFE
plugin is enabled.
In such a case, starting the service triggers a call to `docker-compose stop`
in the local context. This call fails because the "learning" service does not
exist in the local context. Note that this issue only seems to occur with
docker-compose v1.
To resolve this issue, we create two additional filters for
the dev context, which emulate the behaviour of the local context. With this approach, we convert the -m/--mount arguments right after they are parsed. Because they are parsed just once, we can get rid of the de-duplication logic initially introduced with the COMPOSE_CLI_MOUNTS context.
Close #711. Close also https://github.com/overhangio/tutor-mfe/issues/57.
Whenever Tutor executes a shell command, it logs out said
command in order to aid in end user understanding/debugging.
In some cases (notably, when running jobs in containers)
the logged command was not accurately quoted. The command
was run correctly, because it was passed in pieces to
``subprocess.Popen``, which correctly joins the pieces together
into a valid POSIX shell command; however, the logged version
of the command was constructed by simply joining the pieces
with spaces. This usually works, but breaks down when running
complex shell commands with nested quoting.
This commit changes the logging to use ``shlex.join``, which
joins command pieces together in a POSIX-compliant way,
presumably the same way as ``subprocess.Popen``.
Example:
tutor local importdemocourse
runs the shell command:
docker-compose -f /home/kyle/.local/share/tutor/env/local/docker-compose.yml -f /home/kyle/.local/share/tutor/env/local/docker-compose.prod.yml -f /home/kyle/.local/share/tutor/env/local/docker-compose.tmp.yml --project-name tutor_local -f /home/kyle/.local/share/tutor/env/local/docker-compose.jobs.yml -f /home/kyle/.local/share/tutor/env/local/docker-compose.jobs.tmp.yml run --rm cms-job sh -e -c 'echo "Loading settings $DJANGO_SE... (several more script lines) ...eindex_course --all --setup'
but the logged shell command was:
docker-compose -f /home/kyle/.local/share/tutor/env/local/docker-compose.yml -f /home/kyle/.local/share/tutor/env/local/docker-compose.prod.yml -f /home/kyle/.local/share/tutor/env/local/docker-compose.tmp.yml --project-name tutor_local -f /home/kyle/.local/share/tutor/env/local/docker-compose.jobs.yml -f /home/kyle/.local/share/tutor/env/local/docker-compose.jobs.tmp.yml run --rm cms-job sh -e -c echo "Loading settings $DJANGO_SE... (several more script lines) ...eindex_course --all --setup
which will not run if copied and pasted back into the
user's terminal, as the importdemocourse shell script is unquoted.
When waiting for pods, it's possible that the deployment may be
complete but, because other pods may have been Evicted or Killed, the
wait wait condition completes.
In certain code paths, such as in `tutor local quickstart`,
`process_mount_points` is called more than once in the same process,
causing mounts to be added to `COMPOSE_LOCAL[_JOBS]_TMP` redundantly.
As a result, docker-compose[.jobs].tmp.yml was occasionally being
rendered with duplicate volume specifiers. Some versions of Docker
Compose ignored this; other versions warned or threw an error.
In order to make `process_mount_points` tolerant to being called
multiple times, we wrap its volume-adding callbacks within a new
hooks context. This allows us to clear said hooks context every
time `process_mount_points` is called, essentially making the
function idempotent.
Co-authored-by: Régis Behmo <regis@behmo.com>
- [Bugfix] Build openedx-dev Docker image even when the host user is root, for instance on Windows. (by @regisb)
- [Bugfix] Patch nutmeg.1 release with [LTI 1.3 fix](https://github.com/openedx/edx-platform/pull/30716). (by @ormsbee)
- [Improvement] Make it possible to override k8s resources in plugins using `k8s-override` patch. (by @foadlind)
Sometimes, the host user is root: this may happen when tutor is run with
"sudo" (which is not recommended) or on Windows. In such cases, building
the image should not fail, but default to a reasonable user. Also, when
we pass an invalid APP_USER_ID as a build arg, then we should fail with
an explicit message.
See this conversation:
https://discuss.overhang.io/t/problem-with-dev-image-build-useradd-uid-0-is-not-unique/2406
The pymongo dependency for edx-platform was updated (3.10.1 to 3.12.3)
in https://github.com/openedx/edx-platform/pull/30569
This caused the following error when running the edx-platform database
migration split_modulestore_django.0002_data_migration as part of
`tutor dev quickstart`:
pymongo.errors.ServerSelectionTimeoutError: client is configured to
connect to a replica set named '' but this node belongs to a set named
'None', Timeout: 30s, Topology Description: <TopologyDescription id:
62bdbaf182687350acf1aeec, topology_type: Single, servers:
[<ServerDescription ('mongodb', 27017) server_type: Unknown, rtt:
None, error=ConfigurationError("client is configured to connect to a
replica set named '' but this node belongs to a set named 'None'")>]>
This commit explicitly sets replicaSet to None to indicate that it's a
standalone MongoDB instance. I also had to remove the CONTENTSTORE entry
from auth.yml because edx-platform's devstack.py assumes it has a
non-null value (set in common.py), and devstack.py executes before
tutor's development.py can set this replicaSet value.
Currently there is no way for plugins to customize Kubernetes resources
defined in Tutor deployment manifests.
This change makes that possible by taking advantage of the strategic
merge patching mechanism in `kustomization.yml`.
Any resource definition in a `k8s-override` patch in a plugin will
override the resource defined by Tutor, provided that their names match.
Reference: https://github.com/overhangio/tutor/pull/675
- [Bugfix] Update problem with hint template so it works with newer python versions. (by @mariajgrimaldi)
- [Feature] Add default PYTHONBREAKPOINT to openedx/Dockerfile (by @Carlos-Muniz)
- [Bugfix] Fix smtp server port in `cms.yml` which was causing email sending failures in the Studio. (by @regisb)
- [Bugfix] Skip waiting for MongoDB if it is served using SRV records. (by @gabor-boros)
- [Improvement] Use `git am` instead of `cherry-pick` to simplify patching process.
- [Improvement] Tutor is now compatible with Docker Compose subcommand.
PYTHONBREAKPOINT has been exposed as an environment variable in
the openedx Dockerfile available to be changed in config.yml. The docs have also been changed to recommend using
breakpoint and explaining how PYTHONBREAKPOINT can be modified to use a
custom debugger.
Close https://github.com/overhangio/2u-tutor-adoption/issues/45
Incorrect format of cms.yml config file was causing the following error on course import:
cms-worker_1 | Traceback (most recent call last):
cms-worker_1 | File "/openedx/edx-platform/cms/djangoapps/cms_user_tasks/tasks.py", line 53, in send_task_complete_email
cms-worker_1 | mail.send_mail(subject, message, from_address, [dest_addr], fail_silently=False)
cms-worker_1 | File "/openedx/venv/lib/python3.8/site-packages/django/core/mail/__init__.py", line 61, in send_mail
cms-worker_1 | return mail.send()
cms-worker_1 | File "/openedx/venv/lib/python3.8/site-packages/django/core/mail/message.py", line 284, in send
cms-worker_1 | return self.get_connection(fail_silently).send_messages([self])
cms-worker_1 | File "/openedx/venv/lib/python3.8/site-packages/django/core/mail/backends/smtp.py", line 102, in send_messages
cms-worker_1 | new_conn_created = self.open()
cms-worker_1 | File "/openedx/venv/lib/python3.8/site-packages/django/core/mail/backends/smtp.py", line 62, in open
cms-worker_1 | self.connection = self.connection_class(self.host, self.port, **connection_params)
cms-worker_1 | File "/opt/pyenv/versions/3.8.12/lib/python3.8/smtplib.py", line 255, in __init__
cms-worker_1 | (code, msg) = self.connect(host, port)
cms-worker_1 | File "/opt/pyenv/versions/3.8.12/lib/python3.8/smtplib.py", line 339, in connect
cms-worker_1 | self.sock = self._get_socket(host, port, self.timeout)
cms-worker_1 | File "/opt/pyenv/versions/3.8.12/lib/python3.8/smtplib.py", line 310, in _get_socket
cms-worker_1 | return socket.create_connection((host, port), timeout,
cms-worker_1 | File "/opt/pyenv/versions/3.8.12/lib/python3.8/socket.py", line 787, in create_connection
cms-worker_1 | for res in getaddrinfo(host, port, 0, SOCK_STREAM):
cms-worker_1 | File "/opt/pyenv/versions/3.8.12/lib/python3.8/socket.py", line 918, in getaddrinfo
cms-worker_1 | for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
cms-worker_1 | socket.gaierror: [Errno -8] Servname not supported for ai_socktype
The reason was that the trailing comma "," was interpreted as being part of the email port.
- 💥 [Feature] Upgrade to Nutmeg: (by @regisb)
- 💥 [Feature] Persistent grades are now enabled by default.
- [Bugfix] Remove edX references from bulk emails ([issue](https://github.com/openedx/build-test-release-wg/issues/100)).
- [Improvement] For Tutor Nightly (and only Nightly), official plugins are now installed from their nightly branches on GitHub instead of a version range on PyPI. This will allow Nightly users to install all official plugins by running ``pip install -e ".[full]"``.
- [Bugfix] Start MongoDB when running migrations, because a new data migration fails if MongoDB is not running
Celery workers failed to start in development with the following stacktrace:
cms-worker_1 | Traceback (most recent call last):
cms-worker_1 | File "/openedx/venv/bin/celery", line 8, in <module>
cms-worker_1 | sys.exit(main())
cms-worker_1 | File "/openedx/venv/lib/python3.8/site-packages/celery/__main__.py", line 16, in main
cms-worker_1 | _main()
cms-worker_1 | File "/openedx/venv/lib/python3.8/site-packages/celery/bin/celery.py", line 322, in main
cms-worker_1 | cmd.execute_from_commandline(argv)
cms-worker_1 | File "/openedx/venv/lib/python3.8/site-packages/celery/bin/celery.py", line 499, in execute_from_commandline
cms-worker_1 | super(CeleryCommand, self).execute_from_commandline(argv)))
cms-worker_1 | File "/openedx/venv/lib/python3.8/site-packages/celery/bin/base.py", line 305, in execute_from_commandline
cms-worker_1 | return self.handle_argv(self.prog_name, argv[1:])
cms-worker_1 | File "/openedx/venv/lib/python3.8/site-packages/celery/bin/celery.py", line 491, in handle_argv
cms-worker_1 | return self.execute(command, argv)
cms-worker_1 | File "/openedx/venv/lib/python3.8/site-packages/celery/bin/celery.py", line 415, in execute
cms-worker_1 | return cls(
cms-worker_1 | File "/openedx/venv/lib/python3.8/site-packages/celery/bin/worker.py", line 221, in run_from_argv
cms-worker_1 | *self.parse_options(prog_name, argv, command))
cms-worker_1 | File "/openedx/venv/lib/python3.8/site-packages/celery/bin/base.py", line 428, in parse_options
cms-worker_1 | self.parser = self.create_parser(prog_name, command)
cms-worker_1 | File "/openedx/venv/lib/python3.8/site-packages/celery/bin/base.py", line 440, in create_parser
cms-worker_1 | description=self._format_description(self.description),
cms-worker_1 | File "/openedx/venv/lib/python3.8/site-packages/celery/bin/base.py", line 462, in _format_description
cms-worker_1 | text.fill_paragraphs(text.dedent(description), width))
cms-worker_1 | File "/openedx/venv/lib/python3.8/site-packages/celery/utils/text.py", line 58, in fill_paragraphs
cms-worker_1 | return sep.join(fill(p, width) for p in s.split(sep))
cms-worker_1 | File "/openedx/venv/lib/python3.8/site-packages/celery/utils/text.py", line 58, in <genexpr>
cms-worker_1 | return sep.join(fill(p, width) for p in s.split(sep))
cms-worker_1 | File "/opt/pyenv/versions/3.8.12/lib/python3.8/textwrap.py", line 391, in fill
cms-worker_1 | return w.fill(text)
cms-worker_1 | File "/opt/pyenv/versions/3.8.12/lib/python3.8/textwrap.py", line 363, in fill
cms-worker_1 | return "\n".join(self.wrap(text))
cms-worker_1 | File "/opt/pyenv/versions/3.8.12/lib/python3.8/textwrap.py", line 354, in wrap
cms-worker_1 | return self._wrap_chunks(chunks)
cms-worker_1 | File "/opt/pyenv/versions/3.8.12/lib/python3.8/textwrap.py", line 248, in _wrap_chunks
cms-worker_1 | raise ValueError("invalid width %r (must be > 0)" % self.width)
cms-worker_1 | ValueError: invalid width -2 (must be > 0)
This issue was reported upstream here: https://github.com/celery/celery/issues/6302
It is caused by the `tty: true` statement, for some reason. It will be fixed in
Nutmeg, after celery is upgraded to 5.2.6.
Close #681.
I noticed `pip uninstall -y tutor` will not uninstall the plugins, so I made this PR. I know it's ugly, but I don't find any other way of doing it. Let me know if there are better choices 😊
- [Security] Apply logout redirect url security fix. (by @regisb)
- [Feature] Make it possible to force the rendering of a given template, even when the template path matches an ignore pattern. (by @regisb)
- 💥[Fix] Get rid of the `tutor config render` command, which is useless now that themes can be implemented as plugins. (by @regisb)
- [Security] Apply logout redirect url security fix. (by @regisb)
- [Feature] Make it possible to force the rendering of a given template, even when the template path matches an ignore pattern. (by @regisb)
- 💥[Fix] Get rid of the `tutor config render` command, which is useless now that themes can be implemented as plugins. (by @regisb)
When rendering theme files in a plugin, the *.scss files are stored in a
"partials" subdirectory, which was ignored by the environment rendering logic.
To render these files, we move the path ignoring logic to a filter, which is a
list of regular expressions. Values in this filter can be overridden by another
filter.
See the corresponding issue in the indigo theme plugin:
https://github.com/overhangio/tutor-indigo/issues/24
- [Fix] Truncate site display name to 50 characters with a warning, fixing data too long error for long site names. (by @navinkarkera)
- [Feature] Add patch to allow overriding final openedx docker image CMD.
- [Fix] Ignore Python plugins that cannot be loaded. (by @regisb)
- [Improvement] Faster and more reliable builds with `npm clean-install` instead of `npm install`. (by @regisb. Thanks @ghassanmas!)
- [Fix] Fix 500 error during studio login. (by @regisb)
- [Fix] Fix updates for the Caddy deployment in multi-node Kubernetes clusters (#660). Previously, Caddy configuration updates might fail if the Kubernetes cluster had more than one worker node. (by @fghaas)
When running multiple concurrent versions of a plugin there are sometimes
version conflicts that prevent the plugin from being loaded. Prior to v1, Tutor
was correctly ignoring plugins that could not be loaded. During the transition
to v1 we lost that feature because we only captured TutorErrors.
Now that Tutor is the official community installation for Open edX, it no
longer makes sense to host a forum that is separate from the general Open edX
forum. Moving conversations there will encourage cross-communication between
projects and maintainers. This change is part of a larger overhaul described in
this Tutor enhancement proposal (TEP):
https://discuss.overhang.io/t/tep-rethinking-the-tutor-maintainers-program/2724
In the future, plugin maintainers should point their users to the Open edX
forum as well. They are encouraged to create dedicated "tutor-pluginnname" tags
on the forum and to set their notification level to "watching".
The default user in the mysql container is 'mysql',
so the `mysql` command tries to use the 'mysql' MySQL user by default.
But, in the MySQL dump instructions,
we are providing the MySQL root user's password, so we need
to specify `MYSQL_ROOT_USERNAME` as the MySQL user when invoking `mysql`.
When a Pod associated with a Deployment is updated (for example, due
to a change to its ConfigMap, or an updated image reference),
Kubernetes uses a ReplicaSet to spin up a Pod with the new
configuration, and once it is up, it tears down the old one.
In case of the Caddy Deployment, this is complicated by the fact that
it uses a Persistent Volume Claim (PVC), whose corresponding volume
uses a Read/Write-Once (RWO) configuration. This means that it can
only be used by multiple Pods if all those Pods all run on the same
Kubernetes worker node.
In order to enable rolling upgrades for the Caddy Deployment, we need
to ensure that its replacement Pod is scheduled on the same node as
the original Pod.
Thus, add a pod affinity rule that will force exactly that behavior.
Reference:
https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/
The other Tutor services that use volumes (MySQL, Redis, Elasticsearch
and MongoDB) do not need this fix, since they all use the "Recreate"
deployment strategy: their Pods are all automatically torn down before
being replaced. This strategy is not needed for Caddy, and using a pod
affinity rule is less disruptive to the learner experience.
Commit 514e3fce22 made it so
that dev containers were to load mounts from
env/dev/docker-compose.tmp.yml. However, it did not update
the code to generates the docker-compose.tmp.yml files.
This manifested as mounts simply not working in dev mode.
Additionally, we make the docker-compose.jobs.tmp.yml files
follow the same local vs dev differentiation that
was introduced in 514e3fce22.
With this change, we want to better highlight the contributions of
developers to Tutor. We want to publicly acknowledge the positive impact
that individuals and companies have on the development of the platform.
to that end, each changelog entry can now be suffixed with the name of
the author (individual or company) who authored the change. These names
will find their way to the release notes for every release. Eventually,
we also want to spread these release notes more widely. For instance, we
could post new releases to the forum to notify the community of
important changes.
If you have contributed to Tutor in the past, feel free to open a PR and
append your name to the changes that you made. We will not be able to
update the release notes for every release out there, but your
contributions will be acknowledged from the changelog.
When mounting a directory in a dev-only container, such as the
"learning" mfe, docker-compose is failing because it is attempting to
run "docker-compose stop" in the local context -- which knows nothing
about the learning container.
To resolve this, we store tmp volumes either in the local or dev
docker-compose.yml, and load either one depending on the context.
- [Improvement] Add the `COMPOSE_PROJECT_STARTED` action and run `dev
stop` on `local start` (and vice versa).
- [Feature] Introduce `local/dev copyfrom` command to copy contents from
a container.
- [Bugfix] Fix a race condition that could prevent a newly provisioned
LMS container from starting due to a `FileExistsError` when creating
data folders.
- [Deprecation] Mark `tutor dev runserver` as deprecated in favor of
`tutor dev start`. Since `start` now supports bind-mounting and
breakpoint debugging, `runserver` is redundant and will be removed in a
future release.
- [Improvement] Allow breakpoint debugging when attached to a service
via `tutor dev start SERVICE`.
- [Security] Apply rate limiting security fix (see
[commit](b5723e416e)).
- [Feature] Introduce the ``-m/--mount`` option in ``local`` and ``dev``
commands to auto-magically bind-mount folders from the host.
- [Feature] Add `tutor dev quickstart` command, which is similar to
`tutor local quickstart`, except that it uses dev containers instead
of local production ones and includes some other small differences for
the convience of Open edX developers. This should remove some friction
from the Open edX development setup process, which previously required
that users provision using local producation containers (`tutor local
quickstart`) but then stop them and switch to dev containers (`tutor
local stop && tutor dev start -d`).
- 💥[Improvement] Make it possible to run `tutor k8s exec <command with
multiple arguments>` (#636). As a consequence, it is no longer
possible to run quoted commands: `tutor k8s exec "<some command>"`.
Instead, you should remove the quotes: `tutor k8s exec <some command>`.
- 💥[Deprecation] Drop support for the `TUTOR_EDX_PLATFORM_SETTINGS`
environment variable. It is now recommended to create a plugin
instead.
- 💥[Improvement] Complete overhaul of the plugin extension mechanism.
Tutor now has a hook-based Python API: actions can be triggered at
different points of the application life cycle and data can be modified
thanks to custom filters. The v0 plugin API is still supported, for
backward compatibility, but plugin developers are encouraged to migrate
their plugins to the new API. See the new plugin tutorial for more
information.
- [Improvement] Improved the output of `tutor plugins list`.
- [Feature] Add `tutor [dev|local|k8s] status` command, which provides
basic information about the platform's status.
Github release CI was running on ubuntu 18.04 withh python 3.6.
Installing tomli==2.0.1, which is required in dev, triggers a failure in
python 3.6 because it is no longer available.
Github release CI was running on ubuntu 18.04 withh python 3.6.
Installing tomli==2.0.1, which is required in dev, triggers a failure in
python 3.6 because it is no longer available.
- [Improvement] Add the `COMPOSE_PROJECT_STARTED` action and run `dev
stop` on `local start` (and vice versa).
- [Feature] Introduce `local/dev copyfrom` command to copy contents from
a container.
- [Bugfix] Fix a race condition that could prevent a newly provisioned
LMS container from starting due to a `FileExistsError` when creating
data folders.
- [Deprecation] Mark `tutor dev runserver` as deprecated in favor of
`tutor dev start`. Since `start` now supports bind-mounting and
breakpoint debugging, `runserver` is redundant and will be removed in a
future release.
- [Improvement] Allow breakpoint debugging when attached to a service
via `tutor dev start SERVICE`.
- [Security] Apply rate limiting security fix (see
[commit](b5723e416e)).
- [Feature] Introduce the ``-m/--mount`` option in ``local`` and ``dev``
commands to auto-magically bind-mount folders from the host.
- [Feature] Add `tutor dev quickstart` command, which is similar to
`tutor local quickstart`, except that it uses dev containers instead
of local production ones and includes some other small differences for
the convience of Open edX developers. This should remove some friction
from the Open edX development setup process, which previously required
that users provision using local producation containers (`tutor local
quickstart`) but then stop them and switch to dev containers (`tutor
local stop && tutor dev start -d`).
- 💥[Improvement] Make it possible to run `tutor k8s exec <command with
multiple arguments>` (#636). As a consequence, it is no longer
possible to run quoted commands: `tutor k8s exec "<some command>"`.
Instead, you should remove the quotes: `tutor k8s exec <some command>`.
- 💥[Deprecation] Drop support for the `TUTOR_EDX_PLATFORM_SETTINGS`
environment variable. It is now recommended to create a plugin
instead.
- 💥[Improvement] Complete overhaul of the plugin extension mechanism.
Tutor now has a hook-based Python API: actions can be triggered at
different points of the application life cycle and data can be modified
thanks to custom filters. The v0 plugin API is still supported, for
backward compatibility, but plugin developers are encouraged to migrate
their plugins to the new API. See the new plugin tutorial for more
information.
- [Improvement] Improved the output of `tutor plugins list`.
- [Feature] Add `tutor [dev|local|k8s] status` command, which provides
basic information about the platform's status.
Running `local start` while a dev platform is still running is a common sourse
of mistakes. Here we introduce a new action to automatically stop local and dev
projects whenever a project with a different name is started.
`copyfrom` copies data from a container to the local filesystem. It's similar
to bindmount, but less clunky, and more intuitive. Also, it plays along great
with `--mount`. Eventually we'll just get rid of the `bindmount` command and
the `--volume` option.
Previously, `tutor dev runserver --volume=x --volume=y`
would log:
'runserver' is deprecated and will be removed in a future release.
Use 'start' instead. Bind-mounts can be specified using '-m/--mount'.
Bind-mounts can be specified using '-m/--mount'.
`tutor dev runserver` will be removed in a future release.
Developers are encouraged to use `tutor dev start` instead,
which is more flexible and provides a consistent interface
with `tutor local start`.
As part of this deprecation, we enable the `tty` and
`stdin_open` options on development docker-compose
services. This will allow developers to use `start`
for breakpoint debugging, which was previously only
availble via `runserver`. Several parallel PRs have
been merged in order to make the same change in the
development services of the official plugins.
Although `start` does not support the `--volume` option,
it supports a more-powerful `--mount` option. So, where
developers previously used:
tutor dev runserver --volume ...
to bind-mount host directories, they should now use:
tutor dev start --mount ...
Resolves https://github.com/overhangio/2u-tutor-adoption/issues/61
When a v1 plugin was installed, several things were happening regarding tests:
1. v1 plugin loading was happening despite the TUTOR_IGNORE_ENTRYPOINT_PLUGINS
environment variable.
2. the CORE_READY event was not triggered because it was happening just once at
import time.
This was causing some tests to incorrectly load the MFE plugin.
The `--mount` option is available both with `tutor local`
and `tutor dev` commands. It allows users to easily bind-mount containers from
the host to containers. Yes, I know, we already provide that possibility with
the `bindmount` command and the `--volume=/path/` option. But these suffer from
the following drawbacks:
- They are difficult to understand.
- The "bindmount" command name does not make much sense.
- It's not convenient to mount an arbitrary folder from the host to multiple
containers, such as the many lms/cms containers (web apps, celery workers and
job runners).
To address this situation, we now recommend to make use of --mount:
1. `--mount=service1[,service2,...]:/host/path:/container/path`: manually mount
`/host/path` to `/container/path` in container "service1" (and "service2").
2. `--mount=/host/path`: use the new v1 plugin API to discover plugins that
will detect this option and select the right containers in which to bind-mount
volumes. This is really nifty...
Close https://github.com/overhangio/2u-tutor-adoption/issues/43
Add `tutor dev quickstart` command, which is equivalent to
`tutor local quickstart`, but uses dev containers instead
of local production ones and includes some other small
differences for the convience of Open edX developers.
This should remove some friction
from the Open edX development setup process, which previously
required that users provision using local producation
containers but then stop them and switch to dev containers:
* tutor local quickstart
* tutor local stop
* tutor dev start -d
Document the command and its improved workflow in
./docs/tutorials/nightly.rst
Fixes overhangio/2u-tutor-adoption#58
The entrypoint in the "openedx" Docker image was used only to define the
DJANGO_SETTINGS_MODULE environment variable, based on SERVICE_VARIANT and
SETTINGS. We ditch SETTINGS in favour of defining explicitely
DJANGO_SETTINGS_MODULE.
The problem with the Docker entrypoint is that it was bypassed whenever we ran
`tutor local exec` or `tutor k8s exec`. By removing it we make it simpler for
end-users to run manage.py commands in kubernetes.
Previously, it was possible to override settings by defining the
TUTOR_EDX_PLATFORM_SETTINGS environment variable. But let's face it:
- It was not very well supported.
- It was poorly explained.
- It was not very useful.
- It causes unnecessary code complexity.
For these reasons, we drop that feature.
This is a very large refactoring which aims at making Tutor both more
extendable and more generic. Historically, the Tutor plugin system was
designed as an ad-hoc solution to allow developers to modify their own
Open edX platforms without having to fork Tutor. The plugin API was
simple, but limited, because of its ad-hoc nature. As a consequence,
there were many things that plugin developers could not do, such as
extending different parts of the CLI or adding custom template filters.
Here, we refactor the whole codebase to make use of a generic plugin
system. This system was inspired by the Wordpress plugin API and the
Open edX "hooks and filters" API. The various components are added to a
small core thanks to a set of actions and filters. Actions are callback
functions that can be triggered at different points of the application
lifecycle. Filters are functions that modify some data. Both actions and
filters are collectively named as "hooks". Hooks can optionally be
created within a certain context, which makes it easier to keep track of
which application created which callback.
This new hooks system allows us to provide a Python API that developers
can use to extend their applications. The API reference is added to the
documentation, along with a new plugin development tutorial.
The plugin v0 API remains supported for backward compatibility of
existing plugins.
Done:
- Do not load commands from plugins which are not enabled.
- Load enabled plugins once on start.
- Implement contexts for actions and filters, which allow us to keep track of
the source of every hook.
- Migrate patches
- Migrate commands
- Migrate plugin detection
- Migrate templates_root
- Migrate config
- Migrate template environment globals and filters
- Migrate hooks to tasks
- Generate hook documentation
- Generate patch reference documentation
- Add the concept of action priority
Close #499.
MySQL 8 drop the support for creating users by executing `GRANT ALL`. This commit splits the user creation and permission granting, therefore the newer MySQL versions are supported too.
MySQL 8 is supported by edx-platform: 1cdb0347c5/playbooks/roles/mysql/tasks/mysql.yml (L93-L98)
PR #619 set the EDX_PLATFORM_VERSION build arg's default to
OPENEDX_COMMON_VERSION. While this works fine for setting a
non-default branch to run edx code from (say, "master"), it may break
if the user sets OPENEDX_COMMON_VERSION to a branch or tag name that
does not exist upstream in repositories *other than*
EDX_PLATFORM_REPOSITORY.
Thus, introduce a separate configuration parameter,
EDX_PLATFORM_VERSION, to match the build arg of the same name. Set its
default to OPENEDX_COMMON_VERSION.
This way, the user can deploy an arbitrarily-named fork of
edx-platform, while retaining the default OPENEDX_COMMON_VERSION
(like, for example "open-release/maple.3") for everything else.
Previously, the `k8s exec` command did not support unknown "--options". This
made it impossible to launch, say, a django shell in the lms container.
While implementing this feature we saw an opportunity to simplify the way jobs
are handled in the k8s commands.
Close #636.
Another related issue is: https://github.com/overhangio/2u-tutor-adoption/issues/52
The docs recommend commands like:
pip install tutor[full]
pip install -e ./tutor[full]
for installing Tutor. These work fine in bash. For zsh,
though, which is now the default on macOS, quotes are
needed, otherwise zsh will interpret the brackets as
special syntax:
pip install "tutor[full]"
pip install -e "./tutor[full]"
Caveat: I have not tested this myself since I don't
own a Mac, but I've read several issue reports to this
effect, such as:
https://github.com/pypa/pipenv/issues/2830#issuecomment-419593199
The full installation will include all the plugins that
come bundled with Tutor stable. This is made possible by
a recent change to Tutor Nightly
(https://github.com/overhangio/tutor/pull/626).
- [Security] Apply SAML security fix.
- [Improvement] In addition to the Docker build arguments
`EDX_PLATFORM_REPOSITORY` and `NPM_REGISTRY`, also support two corresponding
and identically-named `config.yml` values serving the same purpose.
Previously, the only way for Tutor users to use a fork of edx-platform
or a custom NPM registry was to use build args during the image build.
This is suboptimal in the case of automatically building images from
CI pipelines, which may want to auto-detect when an image needs to be
rebuilt based on config.yml changes.
In addition, the EDX_PLATFORM_VERSION build argument can already be
set via a corresponding config.yml parameter (OPENEDX_COMMON_VERSION),
so it's reasonable to follow that precedent and also introduce
config.yml parameters to correspond with the EDX_PLATFORM_REPOSITORY
and NPM_REGISTRY build arguments.
Thus, introduce two new configuration parameters:
- EDX_PLATFORM_REPOSITORY
- NPM_REGISTRY
These parameters can now optionally be used instead of the
aforementioned build args.
For Tutor Nightly (and only Nightly), official plugins are now installed
from their nightly branches on GitHub instead of a version range on
PyPI. This will allow Nightly users to install all official plugins by
running:
pip install -e ".[full]"
Notes:
* We use the syntax `EGG @ git+REPO@nightly` because the
more common syntax of `git+REPO@nightly#egg=EGG` does not work
when supplied to setup.py's extras_require.
* Unlike other plugins, tutor-license is still installed from PyPI,
but without any version constraint. This is because tutor-license
is a simple, closed-source plugin which activates Wizard edition
for subscribers. It should be available in Nightly but doesn't
need to be installed from its own bleeding-edge branch.
* Unlike most nightly commits, this commit should NOT ever be
reflected on master. When it comes time to merge nightly into
master during the release of Nutmeg, this commit will need to
be manually reverted from master.
* Documentation updates have been made separately so that they
can be merged into master.
The version of the nightly python package should not include the "-nightly"
suffix. That's because when we `pip install -e` tutor plugins, pip also
installs the latest tutor release, as part of the requirements. This overrides
the local (nightly) installation of tutor.
See: https://app.slack.com/client/T02SNA1T6/C02V3GHE3UP
Before edx-platform was upgraded to Celery 5, lms-worker and
cms-worker could be invoked using this syntax:
celery worker --app=APP <args> --maxtasksperchild=N <args>
Since the recent Celery 5 upgrade (edx-platform commit 0588c92),
though, this fails with the messages:
You are using `--app` as an option of the worker sub-command:
celery worker --app celeryapp <...>
The support for this usage was removed in Celery 5.0.
Instead you should use `--app` as a global option:
celery --app celeryapp worker <...>
and:
Error: No such option: --maxtasksperchild
(Possible options: --max-memory-per-child, --max-tasks-per-child)
So, this commit changes the lms-worker and cms-worker invocations to:
celery --app=APP <args> --max-tasks-per-child=N <args>
- [Bugfix] Fix dockerize on arm64 by switching to the [powerman/dockerize](https://github.com/powerman/dockerize) fork (#591).
- [Bugfix] Fix "Unexpected args" error during service initialization on Kubernetes (#611).
The version of dockerize that shipped with the "openedx" image was not
compatible with arm64. The original project is unmaintained, but there
is a fork that provides a version that is compatible with arm64.
This was tested on arm64 with buildx:
docker buildx build --tag=openedx --platform=linux/arm64 ~/.local/share/tutor/env/build/openedx
Close #591
- [Bugfix] Fix `local/k8s quickstart` commands when upgrading from an older release (#595).
- [Bugfix] Fix running the default exim-relay SMTP server on arm64 (#600).
- [Feature] Add `tutor k8s apply` comand, which is a direct interface with `kubectl apply`.
- [Feature] Add `openedx-dockerfile-minimal` patch, which you can use to install custom packages and run commands as root in the Docker image.
I found the existing docs a bit light on the particulars
of how the YAML and Python plugin APIs relate. I was
able to figure it out (there's a nice congruence
between them) but I think these tweaks should it make
it more immediately obvious to readers how the Python
API is a essentially a superset of the YAML API that
allows for dynamic behavior.
because it only contains CLI reference information currently.
The folder structure implies that eventually there will be
more reference material, so the name of 'reference.rst'
was *not* changed to 'cli-reference.rst'.
- [Security] Fix vulnerability in call to invalid enrollment API (see [commit](e9369cffde)).
- [Bugfix] Fix "Internal Server Error / AttributeError / object has no attribute 'get_metadata'" in learning MFE.
- [Improvement] Replace all links to github.com/edx by github.com/openedx, following the migration of all repositories.
- [Bugfix] Fix `k8s start caddy` command.
- [Bugfix] Fix authentication in development due to missing SameSite policy on session ID cookie.
- [Bugfix] Display properly themed favicon.ico image in LMS, Studio and microfrontends.
- [Bugfix] Fix "LazyStaticAbsoluteUrl is not JSON serializable" error when sending bulk emails.
- [Bugfix] Fix `tutor local importdemocourse` fails when platform is not up.
In development, it was no longer possible to authenticate to the lms. Ater
signing in, the session ID could not be dropped, and thus the user was not
signed in, although no error was logged -- just a warning in the browser
console.
This problem was caused by the fact that the SameSite policy was set to "None"
in development.
Previously, we were redirecting all /*favicon.ico requests to the default
favicon. This meant that the favicon might not necessarily be correctly themed,
most notably in MFEs. Here, we resolve this issue by redirecting to the
theme-agnostic theming/asset/* url. Also, we restrict the overly generic regexp
for favicon url matching. We verified that we did not miss any url by running
the following command on the demo server:
tutor local logs caddy | grep --only-matching "host.*favicon.ico" | sort | uniq
The LazyStaticAbsoluteUrl object was breaking bulk emails again with the
following stacktrace:
2022-01-11 13:50:10,591 ERROR 12 [celery.app.trace] [user None] [ip None] trace.py:255 - Task lms.djangoapps.instructor_task.tasks.send_bulk_course_email[26b93357-018a-408f-b3f7-b69722447c5b] raised unexpected: EncodeError(TypeError('Object of type LazyStaticAbsoluteUrl is not JSON serializable'))
Traceback (most recent call last):
File "/openedx/venv/lib/python3.8/site-packages/kombu/serialization.py", line 50, in _reraise_errors
yield
File "/openedx/venv/lib/python3.8/site-packages/kombu/serialization.py", line 221, in dumps
payload = encoder(data)
File "/openedx/venv/lib/python3.8/site-packages/kombu/utils/json.py", line 69, in dumps
return _dumps(s, cls=cls or _default_encoder,
File "/openedx/venv/lib/python3.8/site-packages/simplejson/__init__.py", line 398, in dumps
return cls(
File "/openedx/venv/lib/python3.8/site-packages/simplejson/encoder.py", line 296, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/openedx/venv/lib/python3.8/site-packages/simplejson/encoder.py", line 378, in iterencode
return _iterencode(o, 0)
File "/openedx/venv/lib/python3.8/site-packages/kombu/utils/json.py", line 59, in default
return super(JSONEncoder, self).default(o)
File "/openedx/venv/lib/python3.8/site-packages/simplejson/encoder.py", line 272, in default
raise TypeError('Object of type %s is not JSON serializable' %
TypeError: Object of type LazyStaticAbsoluteUrl is not JSON serializable
The point of that lazy object was to link to the lms logo even when a custom
theme was enabled. Luckily, we no longer need this lazy evaluation because we
now have theme-agnostic urls that point to static asset (see
https://github.com/openedx/edx-platform/pull/29461).
See:
https://discuss.overhang.io/t/error-while-sending-bulk-emails-lazystaticabsoluteurl-is-not-json-serializable/2176/
In the LMS/CMS Dockerfile, the env var STUDIO_CFG is set
in order to point CMS at its configuration json/yaml file.
Since https://github.com/edx/edx-platform/pull/29534
(which introduced 0013-cms-vs-studio.rst), the STUDIO_CFG
variable has been deprecated in favor of CMS_CFG.
This change updates the Dockerfile to reflect the new
preferred environment variable.
The only noticeable impact of this change is that it
will remove a depreation warning from Django startup
for tutor uses running off of Open edX master.
upgrading.
- [Bugfix] During upgrade, make sure that environment is up-to-date
prior to prompting to rebuild the custom images.
- [Bugfix] Fix ownership of mysql data, in particular when upgrading a
Kubernetes cluster to Maple.
- [Bugfix] Ensure that ``tutor k8s upgrade`` is run during ``tutor k8s
quickstart``, when necessary.
- 💥[Bugfix] By default, detect the current version during ``tutor
k8s/local upgrade``.
- [Bugfix] Fix upgrading from Lilac to Maple on Kubernetes by deleting
deployments and services.
`upgrade` had several issues, which are summarized here:
https://discuss.overhang.io/t/confusing-instructions-during-upgrade/2281/7
- The docs say that you should run quickstart, but what most people will see is
the big command tutor local upgrade --from=lilac verbatim paragraph.
- The local upgrade command should be very explicit about the fact that users
need to run quickstart.
- Maybe the name of the local upgrade command should be improved.
- When upgrading tutor from one major release to the next, there should be a
more explicit warning to inform users of what they are doing (see this other
conversation 1)
- We should tell people that they almost certainly need to enable the tutor and
the mfe plugins, if they are not enabled during upgrade.
- A link to all of the breaking changes from the changelog should be
prominently displayed during upgrade.
- The docs should emphasize that upgrading from one major release to the next
is potentially a risky endeavor and that downgrading is not possible. The docs
should also link to the changelog.
This commit has grown slightly beyond the intended scope, but the changes should be mostly positive.
In theory, we can assign ownership of mysql data to just any user. But in
Lilac, mysql was running with user 999. When upgrading to Maple, on Kubernetes,
the fsGroupChangePolicy was causing a change of the data *group* (to 1000) but
not of the user. This was causing a crash with the following error:
[ERROR] InnoDB: The error means mysqld does not have the access rights to the directory.
When upgrading from Lilac, all services break with the following error:
Service "***" is invalid: spec.ports[0].nodePort: Forbidden: may not be used when `type` is 'ClusterIP'
Upgrading deployments fails as well:
Deployment.apps "***" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app.kubernetes.io/instance":"openedx-********", "app.kubernetes.io/managed-by":"tutor", "app.kubernetes.io/name":"***", "app.kubernetes.io/part-of":"openedx"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable
That's because deployments and services need to be deleted as part of the Maple
upgrade. So that's what we do as part of `tutor k8s upgrade --from=lilac`. And
we take the opportunity to:
1. Run upgrade as part of quickstart, when necessary.
2. Default to lilac during `tutor k8s upgrade`.
Close #551.
- [Security] Upgrade Django to 3.2.11 in edx-platform.
- [Security] Prevent non-staff users from searching usernames by email by
abusing the logout url.
Without this patch, it is possible to search for account info including
username by using the email of a learner. This fix disallows searching using
email by regular users and restricts this feature to only staff and superusers.
- A shared cookie domain between lms and cms is no longer recommended:
https://github.com/edx/edx-platform/blob/master/docs/guides/studio_oauth.rst
- refactor: clean mounted data folder in lms/cms. In Lilac, the
bind-mounted lms/data and cms/data folders are a mess because new
folders are created there for every new course organisation. These
folders are empty. As far as we know they are useless... With this
change we move these folders to a dedicated "modulestore" subdirectory;
which corresponds better to the initial intent of the fs_root setting.
- fix: frontend failure during login to the lms. See:
https://github.com/openedx/build-test-release-wg/issues/104
- feat: move all forum-related code to a dedicated plugin. Forum is an
optional feature, and as such it deserves its own plugin. Starting from
Maple, users will be able to install the forum from
https://github.com/overhangio/tutor-forum/
- migrate from DCS_* session cookie settings to SESSION_*. That's
because edx-platform no longer depends on django-cookies-samesite. Close
https://github.com/openedx/build-test-release-wg/issues/110
- get rid of tons of deprecation warnings in the lms/cms
- feat: make it possible to point to themed assets. Cherry-picking this
change makes it possible to point to themed assets with a theme-agnostic
url, notably from MFEs.
- Install all official plugins as part of the `tutor[full]` package.
- Don't print error messages about loading plugins during autocompletion.
- Prompt for image building when upgrading from one release to the next.
- Add `tutor local start --skip-build` option to skip building Docker images.
Close #450.
Close #545.
This introduces quite a few changes to make it easier to run Caddy as a load
balancer in Kubernetes:
- Make it possible to start/stop a selection of resources with ``tutor k8s
start/stop [names...]``.
- Make it easy to deploy an independent LoadBalancer by converting the caddy
service to a NodePort when ``ENABLE_WEB_PROXY=false``.
- Add a ``app.kubernetes.io/component: loadbalancer`` label to the LoadBalancer
service.
- Add ``app.kubernetes.io/name`` labels to all services.
- Preserve the LoadBalancer service in ``tutor k8s stop`` commands.
- Wait for the caddy deployment to be ready before running initialisation jobs.
Close #532.
On some providers (notably: DigitalOcean) NodePort services are not exposed to
the outside world. But this is not what the Kubernetes spec describes:
https://kubernetes.io/docs/concepts/services-networking/service/#publishing-services-service-types
Thus, there is a risk that NodePort services are exposed to the outside world
in some context. To avoid this, we convert all NodePort to ClusterIP resources.
Previously, configuration management was very confusing because we kept mixing
"base" and "defaults" configuration:
- It was difficult to make the difference between core settings that were
necessary (e.g: passwords) as opposed to others that could simply be
defaulted to.
- The order of settings in config.yml mattered: config entries that depended on
other needed to be defined later. As a consequence, Tutor was not compatible
with Python 3.5, where dict entries are not sorted.
Python 3.5 has reached end of life in September 3.5. Anyway, Tutor was not
compatible because some dev dependencies, such as astroid 2.8.3, are no longer
available in 3.5.
This means that we can now start using many python 3.6 niceties, such as
f-strings \o/
Through the commonLabels directive in kustomization.yml, all resources
get a label named "app.kubernetes.io/version", which is being set to
the Tutor version at the time of initial deployment.
When the user then subsequently progresses to a new Tutor version,
Kubernetes attempts to update this label — but for Deployment,
ReplicaSet, and DaemonSet resources, this is no longer allowed as of
https://github.com/kubernetes/kubernetes/issues/50808. This causes
"tutor k8s start" (at the "kubectl apply --kustomize" step) to break
with errors such as:
Deployment.apps "redis" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app.kubernetes.io/instance":"openedx-JIONBLbtByCGUYgHgr4tDWu1", "app.kubernetes.io/managed-by":"tutor", "app.kubernetes.io/name":"redis", "app.kubernetes.io/part-of":"openedx", "app.kubernetes.io/version":"12.1.7"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable
Simply removing the app.kubernetes.io/version label from
kustomization.yml will permanently fix this issue for newly created
Kubernetes deployments, which will "survive" any future Tutor version
changes thereafter.
However, *existing* production Open edX deployments will need to throw
the affected Deployments away, and re-create them.
Also, add the Tutor version as a resource annotation instead, using
the commonAnnotations directive.
See also:
https://github.com/kubernetes/client-go/issues/508https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/commonlabels/https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/commonannotations/
Fixes #531.
When nginx was removed in favour of caddy, we decided that plugin
implementations of the "caddyfile" patch should make use of the "port" local
variable. However, local variables are not available from inside plugin
patches, which are rendered outside of the context of the parent templates.
For a more extensive description of the problem, see:
https://github.com/overhangio/tutor-mfe/pull/23#issuecomment-964016190
We still want to make it easy for developers to decide what should the port be
for caddy hosts. To do so, we make use of environment variables that are passed
at runtime to the caddy container.
Thus, a regular plugin patch should look like this:
{{ PLUGIN_HOST }}{$default_site_port} {
import proxy "myplugin:8000"
}
Forum is an optional feature, and as such it deserves its own plugin. Starting
from Maple, users will be able to install the forum from
https://github.com/overhangio/tutor-forum/
Close #450.
In the past, tutor was installed with "pip install tutor-openedx". For
some time (since v12.0.2), "tutor" was installed as a dependency of
"tutor-openedx". Now is the time to get rid of that old package.
The standard way of installing tutor is now with "pip install tutor".
With this change, containers are no longer run as "root" but as unprivileged
users. This is necessary in some environments, notably some Kubernetes
clusters.
To make this possible, we need to manually fix bind-mounted volumes in
docker-compose. This is pretty much equivalent to the behaviour in Kubernetes,
where permissions are fixed at runtime if the volume owner is incorrect. Thus,
we have a consistent behaviour between docker-compose and Kubernetes.
We achieve this by bind-mounting some repos inside "*-permissions" services.
These services run as root user on docker-compose and will fix the required
permissions, as per build/permissions/setowner.sh These services simply do not
run on Kubernetes, where we don't rely on bind-mounted volumes. There, we make
use of Kubernete's built-in volume ownership feature.
With this change, we get rid of the "openedx-dev" Docker image, in the sense
that it no longer has its own Dockerfile. Instead, the dev image is now simply
a different target in the multi-layer openedx Docker image. This makes it much
faster to build the openedx-dev image.
Because we declare the APP_USER_ID in the dev/docker-compose.yml file, we need
to pass the user ID from the host there. The only way to achieve that is with a
tutor config variable. The downside of this approach is that the
dev/docker-compose.yml file is no longer portable from one machine to the next.
We consider that this is not such a big issue, as it affects the development
environment only.
We take this opportunity to replace the base image of the "forum" image. There
is now no need to re-install ruby inside the image. The total image size is
only decreased by 10%, but re-building the image is faster.
In order to run the smtp service as non-root, we switch from namshi/smtp to
devture/exim-relay. This change should be backward-compatible.
Note that the nginx container remains privileged. We could switch to
nginxinc/nginx-unprivileged, but it's probably not worth the effort, as we are
considering to get rid of the nginx container altogether.
Close #323.
This introduces quite a few changes to make it easier to run Caddy as a load
balancer in Kubernetes:
- Make it possible to start/stop a selection of resources with ``tutor k8s
start/stop [names...]``.
- Make it easy to deploy an independent LoadBalancer by converting the caddy
service to a NodePort when ``ENABLE_WEB_PROXY=false``.
- Add a ``app.kubernetes.io/component: loadbalancer`` label to the LoadBalancer
service.
- Add ``app.kubernetes.io/name`` labels to all services.
- Preserve the LoadBalancer service in ``tutor k8s stop`` commands.
- Wait for the caddy deployment to be ready before running initialisation jobs.
Close #532.
On some providers (notably: DigitalOcean) NodePort services are not exposed to
the outside world. But this is not what the Kubernetes spec describes:
https://kubernetes.io/docs/concepts/services-networking/service/#publishing-services-service-types
Thus, there is a risk that NodePort services are exposed to the outside world
in some context. To avoid this, we convert all NodePort to ClusterIP resources.
- 💥[Improvement] Fail on incorrect image name argument in `images
build/pull/push/printtag` commands.
- [Bugfix] Remove trailing slashes in docker-compose files for
[compatibility with docker-compose v2 in
WSL](https://github.com/docker/compose/issues/8558).
- [Improvement] `settheme` now works with preview domain.
- [Feature] Allow specifying extra pip packages through config.yml.
Previously, configuration management was very confusing because we kept mixing
"base" and "defaults" configuration:
- It was difficult to make the difference between core settings that were
necessary (e.g: passwords) as opposed to others that could simply be
defaulted to.
- The order of settings in config.yml mattered: config entries that depended on
other needed to be defined later. As a consequence, Tutor was not compatible
with Python 3.5, where dict entries are not sorted.
Python 3.5 has reached end of life in September 3.5. Anyway, Tutor was not
compatible because some dev dependencies, such as astroid 2.8.3, are no longer
available in 3.5.
This means that we can now start using many python 3.6 niceties, such as
f-strings \o/
Added OPENEDX_EXTRA_PIP_REQUIREMENTS setting, which allows to specify
extra pip packages that should be installed.
Moved "openedx-scorm-xblock" package from Dockerfile to the new setting
in the config.yml.
Through the commonLabels directive in kustomization.yml, all resources
get a label named "app.kubernetes.io/version", which is being set to
the Tutor version at the time of initial deployment.
When the user then subsequently progresses to a new Tutor version,
Kubernetes attempts to update this label — but for Deployment,
ReplicaSet, and DaemonSet resources, this is no longer allowed as of
https://github.com/kubernetes/kubernetes/issues/50808. This causes
"tutor k8s start" (at the "kubectl apply --kustomize" step) to break
with errors such as:
Deployment.apps "redis" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app.kubernetes.io/instance":"openedx-JIONBLbtByCGUYgHgr4tDWu1", "app.kubernetes.io/managed-by":"tutor", "app.kubernetes.io/name":"redis", "app.kubernetes.io/part-of":"openedx", "app.kubernetes.io/version":"12.1.7"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable
Simply removing the app.kubernetes.io/version label from
kustomization.yml will permanently fix this issue for newly created
Kubernetes deployments, which will "survive" any future Tutor version
changes thereafter.
However, *existing* production Open edX deployments will need to throw
the affected Deployments away, and re-create them.
Also, add the Tutor version as a resource annotation instead, using
the commonAnnotations directive.
See also:
https://github.com/kubernetes/client-go/issues/508https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/commonlabels/https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/commonannotations/
Fixes #531.
When running "tutor k8s" we can disable the forum deployment and
service by setting RUN_FORUM to false, but the corresponding job is
added to jobs.yml unconditionally.
Add a conditional to tutor/templates/k8s/jobs.yml so that the forum
job definition is only included when RUN_FORUM is true.
Fixes #525.
When nginx was removed in favour of caddy, we decided that plugin
implementations of the "caddyfile" patch should make use of the "port" local
variable. However, local variables are not available from inside plugin
patches, which are rendered outside of the context of the parent templates.
For a more extensive description of the problem, see:
https://github.com/overhangio/tutor-mfe/pull/23#issuecomment-964016190
We still want to make it easy for developers to decide what should the port be
for caddy hosts. To do so, we make use of environment variables that are passed
at runtime to the caddy container.
Thus, a regular plugin patch should look like this:
{{ PLUGIN_HOST }}{$default_site_port} {
import proxy "myplugin:8000"
}
Forum is an optional feature, and as such it deserves its own plugin. Starting
from Maple, users will be able to install the forum from
https://github.com/overhangio/tutor-forum/
Close #450.
In some cases, plugins declare an entry point but cannot be loaded.
This is the case when they depend on a version of tutor that is not the
one that is currently installed. This use case is very frequent when
working on multiple versions at the same time (i.e: right now, while we
are working on the Maple release). In such cases, it's best just to
ignore the plugin entirely rather than having to re-install all plugins
in the virtualenv.
In the past, tutor was installed with "pip install tutor-openedx". For
some time (since v12.0.2), "tutor" was installed as a dependency of
"tutor-openedx". Now is the time to get rid of that old package.
The standard way of installing tutor is now with "pip install tutor".
- [Improvement] Upgrade all services to open-release/lilac.3.
- [Feature] Make it possible to override job configuration in
development: if they exist, `dev/docker-compose.jobs.yml` and
`dev/docker-compose.jobs.override.yml` will be loaded when running jobs.
- [Improvement] Faster `tutor local start` by building only necessary
images.
Previously, job declarations were always loaded from local/docker-compose.yml
and local/docker-compose.jobs.yml. This meant that it was not possible to
override job declarations in dev mode. It is now the case, with
dev/docker-compose.jobs.yml and dev/docker-compose.jobs.override.yml. Neither
of these files exist yet... But who knows? we might need this feature one day.
In any case the code is much cleaner now.
Before, custom `docker_compose_func` arguments had to be passed to job runners.
This was not very elegant. Also, it prevented us from loading custom job files
in development.
Here, we adopt a better object-oriented approach, where context classes are
ordered hierarchically.
This paves the way for loading `dev/docker-compose.jobs.yml` files in `tutor
dev init` commands -- which will be necessary to fix permissions in dev/local
mode.
Previously, we were building all images every time we ran a "local start"
command. This was causing unnecessary rebuild. Here, instead, we make use of
the `docker-compose up --build`. This means that only the required images will
be rebuilt.
Limits the memory chek to the 'local quickstart' command, makes error
handling more accurate and adds warning messages for some conditions.
Also adds a mention of this in troubleshooting.rst.
Adds a check in the 'local' command group that requires at least
4 GB of RAM to be allocated to Docker when running any of the
local subcommands on macOS. This addresses a common issue where
Docker's default setting (2 GB) causes startup to crash with
misleading error messages.
- 💥[Improvement] Change the `settheme` command such that, by default, a custom theme is assigned to the LMS and the CMS, both in production and development mode.
With this change, containers are no longer run as "root" but as unprivileged
users. This is necessary in some environments, notably some Kubernetes
clusters.
To make this possible, we need to manually fix bind-mounted volumes in
docker-compose. This is pretty much equivalent to the behaviour in Kubernetes,
where permissions are fixed at runtime if the volume owner is incorrect. Thus,
we have a consistent behaviour between docker-compose and Kubernetes.
We achieve this by bind-mounting some repos inside "*-permissions" services.
These services run as root user on docker-compose and will fix the required
permissions, as per build/permissions/setowner.sh These services simply do not
run on Kubernetes, where we don't rely on bind-mounted volumes. There, we make
use of Kubernete's built-in volume ownership feature.
With this change, we get rid of the "openedx-dev" Docker image, in the sense
that it no longer has its own Dockerfile. Instead, the dev image is now simply
a different target in the multi-layer openedx Docker image. This makes it much
faster to build the openedx-dev image.
Because we declare the APP_USER_ID in the dev/docker-compose.yml file, we need
to pass the user ID from the host there. The only way to achieve that is with a
tutor config variable. The downside of this approach is that the
dev/docker-compose.yml file is no longer portable from one machine to the next.
We consider that this is not such a big issue, as it affects the development
environment only.
We take this opportunity to replace the base image of the "forum" image. There
is now no need to re-install ruby inside the image. The total image size is
only decreased by 10%, but re-building the image is faster.
In order to run the smtp service as non-root, we switch from namshi/smtp to
devture/exim-relay. This change should be backward-compatible.
Note that the nginx container remains privileged. We could switch to
nginxinc/nginx-unprivileged, but it's probably not worth the effort, as we are
considering to get rid of the nginx container altogether.
Close #323.
Get Tutor to work on the master branches of Open edX. The corresponding images
will have to be rebuilt manually. Note that the process to contribute to the
nightly branch is slightly different from the master branch (see the
instructions from the corresponding tutorial).
In conversations with edX, we learned that the name "edge" had negative
undertones for historical reasons. Thus, we switch to "nightly", which means
pretty much the same thing.
Here, we make it possible to automatically append a suffix to the version and app
name (in the sense of appdirs). This guarantees that a tutor edge project will
not accidentally override another community release.
In addition, we take the opportunity to document the tutor versioning format.
(I've been meaning to do that for a long time)
It is unnecessary to point to CI, or to indicate the doc version. Instead, we
link directly to the source code. Also, we improve the icon colors and general
appearance.
This ensures that any warning generated from compiling the docs is treated as
an error. Also, building the docs is now one of the steps performed in CI.
<rant>I attempted to actually run Tutor with Podman and I was sorely disappointed.
The only reliable source of docs that I found concerning the integration with
docker-compose is this blog post:
https://www.redhat.com/sysadmin/podman-docker-compose
There are no other official docs 😓
1. The instructions given in the blog post don't work out of the box. Launching
the podman service failed altogether on Ubuntu 20.04 and 20.10. It worked on
CentOS 8, but some parameters need to changed, such as the docker socket path.
2. After I got the podman service working, I managed to get an Open edX
platform running with tutor, but with the root user. Then, containers
complained that they could not write data to the bind-mounted volumes. I
attempted to run as a non-root user, and discovered that the podman socket is
only readable by root. This should explain why all commands from that blog post
are prefixed by sudo.
Long story short, I was hoping to update the tutorial. Instead, I'm just moving
it for the sake of better organisation. For the life of me, I do not understand
why some people would want to run Podman instead of Docker. Bad documentation
is an immediate turn-off for me. From my perspective, podman is mostly an
overblown marketina stunt.</rant>
There is too much information in each of the local/k8s/dev docs pages. The
"guides" that are listed in each one of those pages are moved either to "common
tasks" or to a dedicated "tutorials" section. This paves the way for more
comprehensive tutorials, where we describe how to run the latest master
branches of Open edX.
I am well aware that, as they stand, the tutorials are of poor quality and
should be rewritten. This is a task for another day/commit. For now, we only
move the contents to a separate part of the docs.
Also, we should add a "reference" section to the docs, where we add the result
of `tutor <subcommand> --help`.
Previously, the list of domain names to which a theme was assigned had to be
specified manually. Now, the themes are automatically assigned to the LMS and
the CMS, both in development and production modes.
See: https://discuss.overhang.io/t/no-activation-email-errors-logged-on-user-sign-up/1969
A 500 error was being triggered during user registration.
Traceback (most recent call last):
File "/openedx/venv/lib/python3.8/site-packages/django/core/handlers/exception.py", line 34, in inner
response = get_response(request)
File "/openedx/venv/lib/python3.8/site-packages/django/core/handlers/base.py", line 115, in _get_response
response = self.process_exception_by_middleware(e, request)
File "/openedx/venv/lib/python3.8/site-packages/django/core/handlers/base.py", line 113, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/openedx/venv/lib/python3.8/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
return view_func(*args, **kwargs)
File "/openedx/venv/lib/python3.8/site-packages/django/views/generic/base.py", line 71, in view
return self.dispatch(request, *args, **kwargs)
File "/openedx/venv/lib/python3.8/site-packages/django/utils/decorators.py", line 45, in _wrapper
return bound_method(*args, **kwargs)
File "/openedx/venv/lib/python3.8/site-packages/django/utils/decorators.py", line 45, in _wrapper
return bound_method(*args, **kwargs)
File "/openedx/venv/lib/python3.8/site-packages/django/views/decorators/debug.py", line 76, in sensitive_post_parameters_wrapper
return view(request, *args, **kwargs)
File "./openedx/core/djangoapps/user_authn/views/register.py", line 485, in dispatch
return super().dispatch(request, *args, **kwargs)
File "/openedx/venv/lib/python3.8/site-packages/rest_framework/views.py", line 509, in dispatch
response = self.handle_exception(exc)
File "/openedx/venv/lib/python3.8/site-packages/rest_framework/views.py", line 469, in handle_exception
self.raise_uncaught_exception(exc)
File "/openedx/venv/lib/python3.8/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
raise exc
File "/openedx/venv/lib/python3.8/site-packages/rest_framework/views.py", line 506, in dispatch
response = handler(request, *args, **kwargs)
File "/openedx/venv/lib/python3.8/site-packages/django/utils/decorators.py", line 45, in _wrapper
return bound_method(*args, **kwargs)
File "/openedx/venv/lib/python3.8/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
return view_func(*args, **kwargs)
File "/openedx/venv/lib/python3.8/site-packages/django/utils/decorators.py", line 45, in _wrapper
return bound_method(*args, **kwargs)
File "/openedx/venv/lib/python3.8/site-packages/ratelimit/decorators.py", line 24, in _wrapped
return fn(request, *args, **kw)
File "./openedx/core/djangoapps/user_authn/views/register.py", line 529, in post
response, user = self._create_account(request, data)
File "./openedx/core/djangoapps/user_authn/views/register.py", line 572, in _create_account
user = create_account_with_params(request, data)
File "./openedx/core/djangoapps/user_authn/views/register.py", line 236, in create_account_with_params
compose_and_send_activation_email(user, profile, registration)
File "/openedx/edx-platform/common/djangoapps/student/views/management.py", line 214, in compose_and_send_activation_email
send_activation_email.delay(str(msg))
File "/openedx/venv/lib/python3.8/site-packages/edx_ace/serialization.py", line 29, in __str__
return json.dumps(self, cls=MessageEncoder)
File "/opt/pyenv/versions/3.8.6/lib/python3.8/json/__init__.py", line 234, in dumps
return cls(
File "/opt/pyenv/versions/3.8.6/lib/python3.8/json/encoder.py", line 199, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/opt/pyenv/versions/3.8.6/lib/python3.8/json/encoder.py", line 257, in iterencode
return _iterencode(o, 0)
File "/openedx/venv/lib/python3.8/site-packages/edx_ace/serialization.py", line 119, in default
return super().default(o) # pragma: no cover
File "/opt/pyenv/versions/3.8.6/lib/python3.8/json/encoder.py", line 179, in default
raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type LazyStaticAbsoluteUrl is not JSON serializable
2021-09-28 05:21:52,174 ERROR 122 [django.request] [user 11] [ip XY.XY.XY.XY] log.py:222 - Internal Server Error: /api/user/v2/account/registration/
Traceback (most recent call last):
File "/openedx/venv/lib/python3.8/site-packages/django/core/handlers/exception.py", line 34, in inner
response = get_response(request)
File "/openedx/venv/lib/python3.8/site-packages/django/core/handlers/base.py", line 115, in _get_response
response = self.process_exception_by_middleware(e, request)
File "/openedx/venv/lib/python3.8/site-packages/django/core/handlers/base.py", line 113, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/openedx/venv/lib/python3.8/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
return view_func(*args, **kwargs)
File "/openedx/venv/lib/python3.8/site-packages/django/views/generic/base.py", line 71, in view
return self.dispatch(request, *args, **kwargs)
File "/openedx/venv/lib/python3.8/site-packages/django/utils/decorators.py", line 45, in _wrapper
return bound_method(*args, **kwargs)
File "/openedx/venv/lib/python3.8/site-packages/django/utils/decorators.py", line 45, in _wrapper
return bound_method(*args, **kwargs)
File "/openedx/venv/lib/python3.8/site-packages/django/views/decorators/debug.py", line 76, in sensitive_post_parameters_wrapper
return view(request, *args, **kwargs)
File "./openedx/core/djangoapps/user_authn/views/register.py", line 485, in dispatch
return super().dispatch(request, *args, **kwargs)
File "/openedx/venv/lib/python3.8/site-packages/rest_framework/views.py", line 509, in dispatch
response = self.handle_exception(exc)
File "/openedx/venv/lib/python3.8/site-packages/rest_framework/views.py", line 469, in handle_exception
self.raise_uncaught_exception(exc)
File "/openedx/venv/lib/python3.8/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
raise exc
File "/openedx/venv/lib/python3.8/site-packages/rest_framework/views.py", line 506, in dispatch
response = handler(request, *args, **kwargs)
File "/openedx/venv/lib/python3.8/site-packages/django/utils/decorators.py", line 45, in _wrapper
return bound_method(*args, **kwargs)
File "/openedx/venv/lib/python3.8/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
return view_func(*args, **kwargs)
File "/openedx/venv/lib/python3.8/site-packages/django/utils/decorators.py", line 45, in _wrapper
return bound_method(*args, **kwargs)
File "/openedx/venv/lib/python3.8/site-packages/ratelimit/decorators.py", line 24, in _wrapped
return fn(request, *args, **kw)
File "./openedx/core/djangoapps/user_authn/views/register.py", line 529, in post
response, user = self._create_account(request, data)
File "./openedx/core/djangoapps/user_authn/views/register.py", line 572, in _create_account
user = create_account_with_params(request, data)
File "./openedx/core/djangoapps/user_authn/views/register.py", line 236, in create_account_with_params
compose_and_send_activation_email(user, profile, registration)
File "/openedx/edx-platform/common/djangoapps/student/views/management.py", line 214, in compose_and_send_activation_email
send_activation_email.delay(str(msg))
File "/openedx/venv/lib/python3.8/site-packages/edx_ace/serialization.py", line 29, in __str__
return json.dumps(self, cls=MessageEncoder)
File "/opt/pyenv/versions/3.8.6/lib/python3.8/json/__init__.py", line 234, in dumps
return cls(
File "/opt/pyenv/versions/3.8.6/lib/python3.8/json/encoder.py", line 199, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/opt/pyenv/versions/3.8.6/lib/python3.8/json/encoder.py", line 257, in iterencode
return _iterencode(o, 0)
File "/openedx/venv/lib/python3.8/site-packages/edx_ace/serialization.py", line 119, in default
return super().default(o) # pragma: no cover
File "/opt/pyenv/versions/3.8.6/lib/python3.8/json/encoder.py", line 179, in default
raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type LazyStaticAbsoluteUrl is not JSON serializable
The reason for that was that edx-ace could not json-serialize the context to be
passed to the registration email renderer. That was caused by the
LazyStaticAbsoluteUrl object created to address missing logo in registration
email. To make sure that this object is serializable by
edx_ace.serialization.MessageEncoder, we add a trivial .to_json() method to the
LazyStaticAbsoluteUrl class.
This error could (at first) not be reproduced in development, because
AUTOMATIC_AUTH_FOR_TESTING is set to true in devstack settings.
When upgrading mongodb, the mongodb container takes a little while to become
ready. Running the "exec" command thus triggers an error:
docker-compose -f /path/to/env/local/docker-compose.yml -f /path/to/env/local/docker-compose.prod.yml --project-name tutor_local exec mongodb mongo --eval db.adminCommand({ setF
eatureCompatibilityVersion: "4.0" })
MongoDB shell version v4.0.24
connecting to: mongodb://127.0.0.1:27017/?gssapiServiceName=mongodb
2021-06-14T10:53:21.510+0000 E QUERY [js] Error: couldn't connect to server 127.0.0.1:27017, connection attempt failed: SocketException: Error connecting to 127.0.0.1:27017 :: caused by :: Connection refused:
connect@src/mongo/shell/mongo.js:356:17
@(connect):2:6
exception: connect failed
Error: Command failed with status 1: docker-compose -f /path/to/env/local/docker-compose.yml -f /path/to/env/local/docker-compose.prod.yml --project-name tutor_local exec mongodb mongo --eval db.adminCommand({ setFeatureCompatibilityVersion: "4.0" })
We add a "sleep" statement to the upgrade process to ensure that the mongodb
container is available.
Turns out, the authentication mechanism should only be defined if there
is an actual authentication. For now, because of the urgency, we
hardcode this auth_mech to ":scram". We'll add a way to override it if
necessary, in the future.
- [Improvement] Make it easier to run edx-platform unit tests.
- [Bugfix] Fix segmentation fault during `tutor config save` on Mac OS M1 (#473). Thanks @ghassanmas!
- [Bugfix] Fix a bug that prevented connecting to external MongoDB instances.
- [Improvement] Make sure that the logo included in email notifications (including discussion responses) is the same as the site logo.
- [Bugfix] Install IPython directly from pypi instead of installing it from source (the reason it was installed from source is no longer relevant). The effect of this shall speed up the process of building the openedx-dev Docker image.
- [Feature] Add "openedx-dockerfile-post-git-checkout" patch.
- [Improvement] In the "openedx" Docker images, convert git patches to cherry-picks for a cleaner source tree.
- 💥[Feature] Make it possible to override local job configuration. This deprecates the older model for running jobs which dates back from a long time ago.
It should be unnecessary to build a custom openedx-dev Docker image. All tests
can run from within the dev Docker image, with a couple additional environment
variables.
Previously, the logo included in emails was loaded from edX' CDN. Here, we make
sure that the logo is actually the same as the site logo. Because the logo may
be theme-specific, we need to compute the logo url at runtime, and use a
lazily-evaluated string.
Close #447.
Change the source of installation Ipyton, install it from PyPy instead of source, which shall speed up the process of building the docker image for openedx-dev
With "git patch", the resulting source tree was dirty, showing uncommitted
changes. Here, we replace "git patch" with "git cherry-pick". We avoid pulling
the entire remote repo by fetching individual commits. To do that, we need to
assign an identity to the git user.
- [Improvement] Upgrade all services to open-release/lilac.2.
- [Bugfix] Fix "`sh` is not a recognized command" in some plugins, including minio.
- [Improvement] Set the default contact mailing email address
- [Bugfix] Fix minio initialisation in Kubernetes.
- [Bugfix] Fix "Invalid command argument" during upgrade from Koa to Lilac.
- [Bugfix] Fix mysql initialisation in docker-compose==2.0.0beta4.
- [Improvement] Tutor is now published on pypi as "tutor".
Previously, the tutor-openedx package was loading tons of template data from
the MANIFEST.in. Turns out, we cannot ignore the MANIFEST.in file with
setuptools. So we need to move tutor-openedx to a separate, dedicated folder.
To auto-discover the package version, we copy it at runtime (in the make
command).
An issue with the latest release of docker-compose was reported here:
https://discuss.overhang.io/t/undefined-entrypoint-throws-error-in-docker-compose-2-0-0-beta-4/1716
The mysql-job definition had an empty entrypoint (`[]`). This was causing the following error:
the initiation of mysql fails with “services.mysql-job.entrypoint must be a string …
Error: Command failed with status 15”
I can't remember at all why we had to define an empty entrypoint. It probably
has to do with the fact that we could not run `sh -e -c "..."` commands in
mysql jobs. Similarly, the k8s job definition sets `command: []`. I tested both
local and k8s deployments without these definitions and they work just fine. So
I guess we can get rid of them.
The package maintainer of the "tutor" package was kind enough to
transfer ownership of the project to us. This is great, because we no
longer have to use the "openedx" suffix, which is trademarked.
For the time being, we keep maintaining the "tutor-openedx" package
which has a 1-to-1 dependency on the "tutor" package. In the future, we
expect that we will no longer push upgrades to tutor-openedx.
Here we add to the docs a few shameless plugs about Cairn -- because
it's really awesome!
We also add a few improvements to the wording, here and there.
- [Bugfix] Fix double pulling mongodb image when upgrading from Koa to Lilac.
- [Improvement] Better logging during `plugins disable`.
- [Bugfix] Fix "upstream sent too big header" error during login of existing users after a Koa to Lilac upgrade.
- [Feature] Added the ability to skip `config.yml` file modification while running `tutor config save` command with `-e` or `--env-only` flag.
- [Feature] Add new config value `FORUM_MONGODB_DATABASE` to set the forum database name
In config.yml the new value FORUM_MONGO_DB_DATABASE was added with `cs_comments_service` as default value.
In docker-entrypoint.sh of forum I changed the hardcoded `cs_commecnts_service` with the new config value.
Multiple .yml files changed to handle the new config value.
When disable a plugin that set config entried, such as the minio plugin, tutor was logging the following:
Disabling plugin minio...
Removed config entry OPENEDX_AWS_ACCESS_KEY=openedx
Removed config entry OPENEDX_AWS_SECRET_ACCESS_KEY={{ MINIO_AWS_SECRET_ACCESS_KEY }}
Plugin disabled
The config values were not rendered during printing, which is a shame, because
the whole point of this log line is to warn users of passwords/secrets that are
being removed. Here, we make sure that the config values are properly rendered.
The new logs are now:
Disabling plugin minio...
Removing config entry OPENEDX_AWS_ACCESS_KEY=openedx
Removing config entry OPENEDX_AWS_SECRET_ACCESS_KEY=64vpCVLxhDxBuNjakSrX4CQg
Plugin disabled
<!-- Are you quite sure that you followed the instructions from the Troubleshooting section in the Tutor documentation? https://docs.tutor.overhang.io/troubleshooting.html -->
<!-- Are you quite sure that you followed the instructions from the Troubleshooting section in the Tutor documentation? https://docs.tutor.edly.io/troubleshooting.html -->
<!-- If not, please take the time to read them. -->
Please check the relevant section of the Tutor docs: `https://docs.tutor.overhang.io/tutor.html#contributing <https://docs.tutor.overhang.io/tutor.html#contributing>`__.
Please check the relevant section of the Tutor docs: `https://docs.tutor.edly.io/tutor.html#contributing <https://docs.tutor.edly.io/tutor.html#contributing>`__.
**Tutor** is a docker-based `Open edX <https://openedx.org>`_ distribution, both for production and local development. The goal of Tutor is to make it easy to deploy, customize, upgrade and scale Open edX. Tutor is reliable, fast, extensible, and it is already used by hundreds of Open edX platforms around the world.
Do you need professional assistance setting up or managing your Open edX platform? Overhang.IO provides online support as part of its `Long Term Support (LTS) offering <https://overhang.io/tutor/pricing>`__.
**Tutor** is the official Docker-based `Open edX <https://openedx.org>`_ distribution, both for production and local development. The goal of Tutor is to make it easy to deploy, customise, upgrade and scale Open edX. Tutor is reliable, fast, extensible, and it is already used to deploy hundreds of Open edX platforms around the world.
Do you need professional assistance setting up or managing your Open edX platform? `Edly <https://edly.io>`__ provides online support as part of its `Open edX installation service <https://edly.io/services/open-edx-installation/>`__.
1. Install the `latest stable release <https://github.com/overhangio/tutor/releases>`_ of Tutor
2. Run ``tutor local quickstart``
2. Run ``tutor local launch``
3. You're done!
Documentation
-------------
Extensive documentation is available online: https://docs.tutor.overhang.io/
Extensive documentation is available: https://docs.tutor.edly.io/
Is there a problem?
-------------------
Please follow the instructions from the `troubleshooting section <https://docs.tutor.overhang.io/troubleshooting.html>`__ in the docs.
Please follow the instructions from the `troubleshooting section <https://docs.tutor.edly.io/troubleshooting.html>`__ in the docs.
.._readme_support_start:
Support
-------
To get community support, go to the official discussion forums: https://discuss.overhang.io. For official support, please subscribe to a Long Term Support (LTS) license at https://overhang.io/tutor/pricing.
To get community support, go to the official Open edX discussion forum: https://discuss.openedx.org. For official support, `Edly <https://edly.io>`__ provides professional assistance as part of its `Open edX installation service <https://edly.io/services/open-edx-installation/>`__.
.._readme_support_end:
@ -85,6 +90,11 @@ To get community support, go to the official discussion forums: https://discuss.
Contributing
------------
We welcome contributions to Tutor! To learn how you can contribute, please check the relevant section of the Tutor docs: `https://docs.tutor.overhang.io/tutor.html#contributing <https://docs.tutor.overhang.io/tutor.html#contributing>`__.
We welcome contributions to Tutor! To learn how you can contribute, please check the relevant section of the Tutor docs: `https://docs.tutor.edly.io/tutor.html#contributing <https://docs.tutor.edly.io/tutor.html#contributing>`__.
.._readme_contributing_end:
License
-------
This work is licensed under the terms of the `GNU Affero General Public License (AGPL) <https://github.com/overhangio/tutor/blob/master/LICENSE.txt>`_.
- [Bugfix] Error "'Crypto.PublicKey.RSA.RsaKey object' has no attribute 'dq'" during `tutor config save` was caused by outdated minimum version of the pycryptodome package. To resolve this issue, run `pip install --upgrade pycryptodome`. (by @regisb)
- [Feature] add `CONFIG_INTERACTIVE` action that allows tutor plugins to interact with the configuration at the time of the interactive questionnaire that happens during tutor local launch. (by @Alec4r).
Tutor offers plenty of possibilities for platform customisation out of the box. There are two main ways in which the base Open edX installation can be customized:
Tutor offers plenty of possibilities for platform customisation out of the box. There are two main ways in which the base Open edX installation can be customised:
a. Modifying the Tutor :ref:`configuration parameters <configuration>`.
b. Modifying the :ref:`Open edX docker image <customise>` that runs the Open edX platform.
This section does not cover :ref:`plugin development <plugins>`. For simple changes, such as modifying the ``*.env.json`` files or the edx-platform settings, *you should not fork edx-platform or tutor*! Instead, you should create a simple :ref:`plugin for Tutor <plugins_yaml>`.
This section does not cover :ref:`plugin development <plugins>`. For simple changes, such as modifying the ``*.env.yml`` files or the edx-platform settings, *you should not fork edx-platform or tutor*! Instead, you should create a simple :ref:`plugin for Tutor <plugins_yaml>`.
.._configuration:
Configuration
-------------
With Tutor, all Open edX deployment parameters are stored in a single ``config.yml`` file. This is the file that is generated when you run ``tutor local quickstart`` or ``tutor config save``. To view the content of this file, run::
With Tutor, all Open edX deployment parameters are stored in a single ``config.yml`` file. This is the file that is generated when you run ``tutor local launch`` or ``tutor config save``. To view the content of this file, run::
cat "$(tutor config printroot)/config.yml"
@ -31,7 +31,7 @@ Or from the system environment::
export TUTOR_PARAM1=VALUE1
Once the base configuration is created or updated, the environment is automatically re-generated. The environment is the set of all files required to manage an Open edX platform: Dockerfile, ``lms.env.json``, settings files, etc. You can view the environment files in the ``env`` folder::
Once the base configuration is created or updated, the environment is automatically re-generated. The environment is the set of all files required to manage an Open edX platform: Dockerfile, ``lms.env.yml``, settings files, etc. You can view the environment files in the ``env`` folder::
ls "$(tutor config printroot)/env"
@ -40,9 +40,6 @@ With an up-to-date environment, Tutor is ready to launch an Open edX platform an
These configuration parameters define which image to run for each service. By default, the docker image tag matches the Tutor version it was built with.
This configuration parameter defines the name of the Docker image to run for the lms and cms containers. By default, the Docker image tag matches the Tutor version it was built with.
This configuration parameter defines the name of the Docker image to run the development version of the lms and cms containers. By default, the Docker image tag matches the Tutor version it was built with.
This configuration parameter defines the Docker image to be used for setting file permissions. The default image sets all containers to be run as unprivileged users.
Custom registry
***************
@ -76,22 +112,95 @@ You may want to pull/push images from/to a custom docker registry. For instance,
(the trailing ``/`` is important)
.._openedx_configuration:
Compose
*******
- ``DOCKER_COMPOSE_VERSION`` (default: ``"3.7"``)
This configuration parameter sets the version of Docker Compose to be used to build all containers.
This defines the git repository from which you install Open edX platform code. If you run an Open edX fork with custom patches, set this to your own git repository. You may also override this configuration parameter at build time, by providing a ``--build-arg`` option.
- ``OPENEDX_COMMON_VERSION`` (default: ``"open-release/quince.1"``, or ``master`` in :ref:`nightly <nightly>`)
This defines the default version that will be pulled from all Open edX git repositories.
- ``EDX_PLATFORM_VERSION`` (default: the value of ``OPENEDX_COMMON_VERSION``)
This defines the version that will be pulled from just the Open edX platform git repositories. You may also override this configuration parameter at build time, by providing a ``--build-arg`` option.
- ``OPENEDX_CMS_UWSGI_WORKERS`` (default: ``2``)
- ``OPENEDX_LMS_UWSGI_WORKERS`` (default: ``2``)
By default there are 2 `uwsgi worker processes <https://uwsgi-docs.readthedocs.io/en/latest/Options.html#processes>`__ to serve requests for the LMS and the CMS. However, each workers requires upwards of 500 Mb of RAM. You should reduce this value to 1 if your computer/server does not have enough memory.
By default, there are 2 `uwsgi worker processes <https://uwsgi-docs.readthedocs.io/en/latest/Options.html#processes>`__ to serve requests for the LMS and the CMS. However, each worker requires upwards of 500 Mb of RAM. You should reduce this value to 1 if your computer/server does not have enough memory.
- ``OPENEDX_CELERY_REDIS_DB`` (default: ``0``)
- ``OPENEDX_CACHE_REDIS_DB`` (default: ``1``)
These two configuration parameters define which redis database to use for Open edX cache and celery task.
These two configuration parameters define which Redis database to use for Open edX cache and celery task.
This defines the registry from which you'll be pulling NPM packages when building Docker images. Like ``EDX_PLATFORM_REPOSITORY``, this can be overridden at build time with a ``--build-arg`` option.
- ``OPENEDX_AWS_ACCESS_KEY`` (default: ``""``)
This configuration parameter sets the Django setting ``AWS_ACCESS_KEY_ID`` in edx-platform's LMS, CMS, envs, and production.py for use by the library django-storages with Amazon S3.
This configuration parameter sets the Django setting ``AWS_SECRET_ACCESS_KEY`` in edx-platform's LMS, CMS, envs, and production.py for use by the library django-storages with Amazon S3.
These configuration parameters are rendered into the ``JWT_AUTH`` dictionary with keys ``JWT_AUDIENCE``, ``JWT_ISSUER``, and ``JWT_SECRET_KEY``, respectively. These parameters may be changed in order to create a custom user login for testing purposes.
Vendor services
~~~~~~~~~~~~~~~
@ -99,16 +208,10 @@ Vendor services
Caddy
*****
- ``RUN_CADDY`` (default: ``true``)
- ``CADDY_HTTP_PORT`` (default: ``80``)
- ``ENABLE_WEB_PROXY`` (default: ``true``)
`Caddy <https://caddyserver.com>`__ is a web server used in Tutor as a web proxy for the generation of SSL/TLS certificates at runtime. If ``RUN_CADDY`` is set to ``false`` then we assume that SSL termination does not occur in the Caddy container, and thus the ``caddy`` container is not started.
Nginx
*****
- ``NGINX_HTTP_PORT`` (default: ``80``)
Nginx is used to route web traffic to the various applications and to serve static assets. When ``RUN_CADDY`` is false, the ``NGINX_HTTP_PORT`` is exposed on the host.
`Caddy <https://caddyserver.com>`__ is a web server used in Tutor both as a web proxy and for the generation of SSL/TLS certificates at runtime. Port indicated by ``CADDY_HTTP_PORT`` is exposed on the host, in addition to port 443. If ``ENABLE_WEB_PROXY`` is set to ``false`` then we assume that SSL termination does not occur in the Caddy container and only ``CADDY_HTTP_PORT`` is exposed on the host.
MySQL
*****
@ -119,7 +222,7 @@ MySQL
- ``MYSQL_ROOT_USERNAME`` (default: ``"root"``)
- ``MYSQL_ROOT_PASSWORD`` (default: randomly generated) Note that you are responsible for creating the root user if you are using a managed database.
By default, a running Open edX platform deployed with Tutor includes all necessary 3rd-party services, such as MySQL, MongoDb, etc. But it's also possible to store data on a separate database, such as `Amazon RDS <https://aws.amazon.com/rds/>`_. For instance, to store data on an external MySQL database, set the following configuration::
By default, a running Open edX platform deployed with Tutor includes all necessary 3rd-party services, such as MySQL, MongoDb, etc. But it's also possible to store data on a separate database, such as `Amazon RDS <https://aws.amazon.com/rds/>`_. For instance, to store data on an external MySQL database set the following configuration::
RUN_MYSQL: false
MYSQL_HOST: yourhost
@ -137,15 +240,21 @@ Elasticsearch
- ``ELASTICSEARCH_PORT`` (default: ``9200``)
- ``ELASTICSEARCH_HEAP_SIZE`` (default: ``"1g"``)
Mongodb
MongoDB
*******
- ``RUN_MONGODB`` (default: ``true``)
- ``MONGODB_HOST`` (default: ``"mongodb"``)
- ``MONGODB_DATABASE`` (default: ``"openedx"``)
- ``MONGODB_HOST`` (default: ``"mongodb"``)
- ``MONGODB_PASSWORD`` (default: ``""``)
- ``MONGODB_PORT`` (default: ``27017``)
- ``MONGODB_USERNAME`` (default: ``""``)
- ``MONGODB_PASSWORD`` (default: ``""``)
- ``MONGODB_USE_SSL`` (default: ``false``)
- ``MONGODB_REPLICA_SET`` (default: ``""``)
- ``MONGODB_AUTH_MECHANISM`` (default: ``""``)
- ``MONGODB_AUTH_SOURCE`` (default: ``"admin"``)
Note that most of these settings will have to be modified to connect to a MongoDB cluster that runs separately of Tutor, such as `Atlas <https://www.mongodb.com/atlas>`__. In particular, the authentication source, mechanism and the SSL connection parameters should not be specified as part of the `host URI <https://www.mongodb.com/docs/manual/reference/connection-string/>`__ but as separate Tutor settings. Supported values for ``MONGODB_AUTH_MECHANISM`` are the same as for pymongo (see the `pymongo documentation <https://pymongo.readthedocs.io/en/stable/examples/authentication.html>`__).
Redis
*****
@ -163,7 +272,7 @@ SMTP
- ``RUN_SMTP`` (default: ``true``)
- ``SMTP_HOST`` (default: ``"smtp"``)
- ``SMTP_PORT`` (default: ``25``)
- ``SMTP_PORT`` (default: ``8025``)
- ``SMTP_USERNAME`` (default: ``""``)
- ``SMTP_PASSWORD`` (default: ``""``)
- ``SMTP_USE_TLS`` (default: ``false``)
@ -176,22 +285,40 @@ SSL/TLS certificates for HTTPS access
- ``ENABLE_HTTPS`` (default: ``false``)
By activating this feature, a free SSL/TLS certificate from the `Let's Encrypt <https://letsencrypt.org/>`_ certificate authority will be created for your platform. With this feature, **your platform will no longer be accessible in HTTP**. Calls to http urls will be redirected to https url.
When ``ENABLE_HTTPS`` is ``true``, the whole Open edX platform will be reconfigured to work with "https" URIs. Calls to "http" URIs will be redirected to "https". By default, SSL/TLS certificates will automatically be generated by Tutor (thanks to `Caddy <https://caddyserver.com/>`__) from the `Let's Encrypt <https://letsencrypt.org/>`_ certificate authority.
The following DNS records must exist and point to your server::
LMS_HOST (e.g: myopenedx.com)
preview.LMS_HOST (e.g: preview.myopenedx.com)
PREVIEW_LMS_HOST (e.g: preview.myopenedx.com)
CMS_HOST (e.g: studio.myopenedx.com)
Thus, **this feature will (probably) not work in development** because the DNS records will (probably) not point to your development machine.
The SSL/TLS certificates will automatically be generated and updated by the Caddy proxy server container at runtime. Thus, as of v11.0.0 you no longer have to generate the certificates manually.
If you would like to perform SSL/TLS termination with your own custom certificates, you will have to keep ``ENABLE_HTTPS=true`` and turn off the Caddy load balancing with ``ENABLE_WEB_PROXY=false``. See the corresponding :ref:`tutorial <web_proxy>` for more information.
.._customise:
.._custom_openedx_docker_image:
Kubernetes
~~~~~~~~~~
- ``K8S_NAMESPACE`` (default: ``"openedx"``)
This configuration parameter sets the Kubernetes Namespace.
This configuration parameter sets the Contact Email.
- ``PLATFORM_NAME`` (default: ``"My Open edX"``)
This configuration parameter sets the Platform Name.
Custom Open edX docker image
----------------------------
@ -205,16 +332,16 @@ The following sections describe how to modify various aspects of the docker imag
tutor local stop
The custom image will be used the next time you run ``tutor local quickstart`` or ``tutor local start``. Do not attempt to run ``tutor local restart``! Restarting will not pick up the new image and will continue to use the old image.
The custom image will be used the next time you run ``tutor local launch`` or ``tutor local start``. Do not attempt to run ``tutor local restart``! Restarting will not pick up the new image and will continue to use the old image.
openedx Docker Image build arguments
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
"openedx" Docker image build arguments
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
When building the "openedx" Docker image, it is possible to specify a few `arguments <https://docs.docker.com/engine/reference/builder/#arg>`__:
These arguments can be specified from the command line, `very much like Docker <https://docs.docker.com/engine/reference/commandline/build/#set-build-time-variables---build-arg>`__. For instance::
@ -223,56 +350,27 @@ These arguments can be specified from the command line, `very much like Docker <
Adding custom themes
~~~~~~~~~~~~~~~~~~~~
Comprehensive theming is enabled by default, but only the default theme is compiled. `Indigo <https://github.com/overhangio/indigo>`__ is a better, ready-to-run theme which you can start using today.
To compile your own theme, add it to the ``env/build/openedx/themes/`` folder::
The ``themes`` folder should have the following structure::
openedx/themes/
mycustomtheme1/
cms/
...
lms/
...
mycustomtheme2/
...
Then you must rebuild the openedx Docker image::
tutor images build openedx
Finally, you should enable your theme with the :ref:`settheme command <settheme>`.
See :ref:`the corresponding tutorial <theming>`.
.._custom_extra_xblocks:
Installing extra xblocks and requirements
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Would you like to include custom xblocks, or extra requirements to your Open edX platform? Additional requirements can be added to the ``env/build/openedx/requirements/private.txt`` file. For instance, to include the `polling xblock from Opencraft <https://github.com/open-craft/xblock-poll/>`_::
Would you like to include custom xblocks, or extra requirements to your Open edX platform? Additional requirements can be added to the ``OPENEDX_EXTRA_PIP_REQUIREMENTS`` parameter in the :ref:`config file <configuration>`. For instance, to include the `polling xblock from Opencraft <https://github.com/open-craft/xblock-poll/>`_::
tutor config save --append OPENEDX_EXTRA_PIP_REQUIREMENTS=git+https://github.com/open-craft/xblock-poll.git
Then, the ``openedx`` docker image must be rebuilt::
tutor images build openedx
To install xblocks from a private repository that requires authentication, you must first clone the repository inside the ``openedx/requirements`` folder on the host::
You may want to run your own flavor of edx-platform instead of the `official version <https://github.com/edx/edx-platform/>`_. To do so, you will have to re-build the openedx image with the proper environment variables pointing to your repository and version::
You may want to run your own flavor of edx-platform instead of the `official version <https://github.com/openedx/edx-platform/>`_. To do so, you will have to re-build the openedx image with the proper environment variables pointing to your repository and version::
@ -284,16 +382,16 @@ Note that your edx-platform version must be a fork of the latest release **tag**
If you don't create your fork from this tag, you *will* have important compatibility issues with other services. In particular:
- Do not try to run a fork from an older (pre-Lilac) version of edx-platform: this will simply not work.
- Do not try to run a fork from an older (pre-Quince) version of edx-platform: this will simply not work.
- Do not try to run a fork from the edx-platform master branch: there is a 99% probability that it will fail.
- Do not try to run a fork from the open-release/lilac.master branch: Tutor will attempt to apply security and bug fix patches that might already be included in the open-release/lilac.master but which were not yet applied to the latest release tag. Patch application will thus fail if you base your fork from the open-release/lilac.master branch.
- Do not try to run a fork from the open-release/quince.master branch: Tutor will attempt to apply security and bug fix patches that might already be included in the open-release/quince.master but which were not yet applied to the latest release tag. Patch application will thus fail if you base your fork from the open-release/quince.master branch.
.._i18n:
Adding custom translations
~~~~~~~~~~~~~~~~~~~~~~~~~~
If you are not running Open edX in English, chances are that some strings will not be properly translated. In most cases, this is because not enough contributors have helped translate Open edX in your language. It happens! With Tutor, available translated languages include those that come bundled with `edx-platform <https://github.com/edx/edx-platform/tree/open-release/lilac.master/conf/locale>`__ as well as those from `openedx-i18n <https://github.com/openedx/openedx-i18n/tree/master/edx-platform/locale>`__.
If you are not running Open edX in English (``LANGUAGE_CODE`` default: ``"en"``), chances are that some strings will not be properly translated. In most cases, this is because not enough contributors have helped translate Open edX into your language. It happens! With Tutor, available translated languages include those that come bundled with `edx-platform <https://github.com/openedx/edx-platform/tree/open-release/quince.master/conf/locale>`__ as well as those from `openedx-i18n <https://github.com/openedx/openedx-i18n/tree/master/edx-platform/locale>`__.
Tutor offers a relatively simple mechanism to add custom translations to the openedx Docker image. You should create a folder that corresponds to your language code in the "build/openedx/locale" folder of the Tutor environment. This folder should contain a "LC_MESSAGES" folder. For instance::
@ -314,9 +412,9 @@ Then, add a "django.po" file there that will contain your custom translations::
..warning::
Don't forget to specify the file ``Content-Type`` when adding message strings with non-ASCII characters; otherwise a ``UnicodeDecodeError`` will be raised during compilation.
The "String to translate" part should match *exactly* the string that you would like to translate. You cannot make it up! The best way to find this string is to copy-paste it from the `upstream django.po file for the English language <https://github.com/edx/edx-platform/blob/open-release/lilac.master/conf/locale/en/LC_MESSAGES/django.po>`__.
The "String to translate" part should match *exactly* the string that you would like to translate. You cannot make it up! The best way to find this string is to copy-paste it from the `upstream django.po file for the English language <https://github.com/openedx/edx-platform/blob/open-release/quince.master/conf/locale/en/LC_MESSAGES/django.po>`__.
If you cannot find the string to translate in this file, then it means that you are trying to translate a string that is used in some piece of javascript code. Those strings are stored in a different file named "djangojs.po". You can check it out `in the edx-platform repo as well <https://github.com/edx/edx-platform/blob/open-release/lilac.master/conf/locale/en/LC_MESSAGES/djangojs.po>`__. Your custom javascript strings should also be stored in a "djangojs.po" file that should be placed in the same directory.
If you cannot find the string to translate in this file, then it means that you are trying to translate a string that is used in some piece of javascript code. Those strings are stored in a different file named "djangojs.po". You can check it out `in the edx-platform repo as well <https://github.com/openedx/edx-platform/blob/open-release/quince.master/conf/locale/en/LC_MESSAGES/djangojs.po>`__. Your custom javascript strings should also be stored in a "djangojs.po" file that should be placed in the same directory.
To recap, here is an example. To translate a few strings in French, both from django.po and djangojs.po, we would have the following file hierarchy::
@ -348,9 +446,9 @@ And djangojs.po::
Then you will have to re-build the openedx Docker image::
tutor images build openedx openedx-dev
tutor images build openedx
Beware that this will take a long time! Unfortunately it's difficult to accelerate this process, as translation files need to be compiled prior to collecting the assets. In development it's possible to accelerate the iteration loop -- but that exercise is left to the reader.
Beware that this will take a long time! Unfortunately, it's difficult to accelerate this process, as translation files need to be compiled before collecting the assets. In development it's possible to accelerate the iteration loop -- but that exercise is left to the reader.
Running a different ``openedx`` Docker image
@ -364,4 +462,18 @@ By default, Tutor runs the `overhangio/openedx <https://hub.docker.com/r/overhan
(See the relevant :ref:`configuration parameters <docker_images>`.)
The customised Docker image tag value will then be used by Tutor to run the platform, for instance when running ``tutor local quickstart``.
The customised Docker image tag value will then be used by Tutor to run the platform, for instance when running ``tutor local launch``.
Passing custom docker build options
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You can set a limited set of Docker build options via ``tutor images build`` command. In some situations it might be necessary to tweak the docker build command, ex- setting up build caching using buildkit.
In these situations, you can set ``--docker-arg`` flag in the ``tutor images build`` command. You can set any `supported options <https://docs.docker.com/engine/reference/commandline/build/#options>`_ in the docker build command, For example::
tutor images build openedx \
--build-arg BUILDKIT_INLINE_CACHE=1 \
--docker-arg="--cache-from" \
--docker-arg="docker.io/myusername/openedx:mytag"
This will result in passing the ``--cache-from`` option with the value ``docker.io/myusername/openedx:mytag`` to the docker build command.
In addition to running Open edX in production, Tutor can be used for local development of Open edX. This means that it is possible to hack on Open edX without setting up a Virtual Machine. Essentially, this replaces the devstack provided by edX.
The following commands assume you have previously launched a :ref:`local <local>` Open edX platform. If you have not done so already, you should run::
For detailed explanations on how to work on edx-platform and its dependencies, see the :ref:`edx_platform` tutorial.
tutor local quickstart
.._edx_platform_dev_env:
In order to run the platform in development mode, you **must** answer no ("n") to the question "Are you configuring a production platform".
First-time setup
----------------
Note that the local.overhang.io `domain <https://dnschecker.org/#A/local.overhang.io>`__ and its `subdomains <https://dnschecker.org/#CNAME/studio.local.overhang.io>`__ all point to 127.0.0.1. This is just a domain name that was setup to conveniently access a locally running Open edX platform.
Firstly, either :ref:`install Tutor <install>` (for development against the named releases of Open edX) or :ref:`install Tutor Nightly <nightly>` (for development against Open edX's master branches).
Once the local platform has been configured, you should stop it so that it does not interfere with the development environment::
Then, optionally, tell Tutor to use a local fork of edx-platform::
tutor local stop
tutor mounts add ./edx-platform
Finally, you should build the ``openedx-dev`` docker image::
Then, launch the developer platform setup process::
tutor images build openedx-dev
tutor dev launch
This ``openedx-dev`` development image differs from the ``openedx`` production image:
This will perform several tasks. It will:
- The user that runs inside the container has the same UID as the user on the host, in order to avoid permission problems inside mounted volumes (and in particular in the edx-platform repository).
- Additional python and system requirements are installed for convenient debugging: `ipython <https://ipython.org/>`__, `ipdb <https://pypi.org/project/ipdb/>`__, vim, telnet.
- The edx-platform `development requirements <https://github.com/edx/edx-platform/blob/open-release/lilac.master/requirements/edx/development.in>`__ are installed.
* build the "openedx-dev" Docker image, which is based on the "openedx" production image but is `specialized for developer usage`_ (eventually with your fork),
* stop any existing locally-running Tutor containers,
* disable HTTPS,
* set ``LMS_HOST`` to `local.edly.io <http://local.edly.io>`_ (a convenience domain that simply `points at 127.0.0.1 <https://dnschecker.org/#A/local.edly.io>`_),
* prompt for a platform details (with suitable defaults),
* start LMS, CMS, supporting services, and any plugged-in services,
* ensure databases are created and migrated, and
* run service initialization scripts, such as service user creation and Waffle configuration.
Since the ``openedx-dev`` is based upon the ``openedx`` docker image, it should be re-built every time the ``openedx`` docker image is modified.
Additionally, when a local clone of edx-platform is bind-mounted, it will:
Run a local development webserver
---------------------------------
* re-run setup.py,
* clean-reinstall Node modules, and
* regenerate static assets.
::
Once setup is complete, the platform will be running in the background:
tutor dev runserver lms # Access the lms at http://local.overhang.io:8000
tutor dev runserver cms # Access the cms at http://studio.local.overhang.io:8001
* LMS will be accessible at `http://local.edly.io:8000 <http://local.edly.io:8000>`_.
* CMS will be accessible at `http://studio.local.edly.io:8001 <http://studio.local.edly.io:8001>`_.
* Plugged-in services should be accessible at their documented URLs.
Now, use the ``tutor dev ...`` command-line interface to manage the development environment. Some common commands are described below.
..note::
If you've added your edx-platform to the bind-mounted folders, you can remove at any time by running::
tutor mounts remove ./edx-platform
At any time, check your configuration by running::
tutor mounts list
Read more about bind-mounts :ref:`below <bind_mounts>`.
Stopping the platform
---------------------
To bring down the platform's containers, simply run::
tutor dev stop
Starting the platform back up
-----------------------------
Once first-time setup has been performed with ``launch``, the platform can be started going forward with the lighter-weight ``start -d`` command, which brings up containers *detached* (that is: in the background), but does not perform any initialization tasks::
tutor dev start -d
Or, to start with platform with containers *attached* (that is: in the foreground, the current terminal), omit the ``-d`` flag::
tutor dev start
When running containers attached, stop the platform with ``Ctrl+c``, or switch to detached mode using ``Ctrl+z``.
Finally, the platform can also be started back up with ``launch``. It will take longer than ``start``, but it will ensure that config is applied, databases are provisioned & migrated, plugins are fully initialized, and (if applicable) the bind-mounted edx-platform is set up. Notably, ``launch`` is idempotent, so it is always safe to run it again without risk to data. Including the ``--pullimages`` flag will also ensure that container images are up-to-date::
tutor dev launch --pullimages
Running arbitrary commands
--------------------------
@ -52,46 +99,140 @@ To open a python shell in the LMS or CMS, run::
You can then import edx-platform and django modules and execute python code.
To collect assets, you can use the ``openedx-assets`` command that ships with Tutor::
To rebuild assets, you can use the ``openedx-assets`` command that ships with Tutor::
tutor dev run lms openedx-assets build --env=dev
.._specialized for developer usage:
Rebuilding the openedx-dev image
--------------------------------
The ``openedx-dev`` Docker image is based on the same ``openedx`` image used by ``tutor local ...`` to run LMS and CMS. However, it has a few differences to make it more convenient for developers:
- The user that runs inside the container has the same UID as the user on the host, to avoid permission problems inside mounted volumes (and in particular in the edx-platform repository).
- Additional Python and system requirements are installed for convenient debugging: `ipython <https://ipython.org/>`__, `ipdb <https://pypi.org/project/ipdb/>`__, vim, telnet.
- The edx-platform `development requirements <https://github.com/openedx/edx-platform/blob/open-release/quince.master/requirements/edx/development.in>`__ are installed.
If you are using a custom ``openedx`` image, then you will need to rebuild ``openedx-dev`` every time you modify ``openedx``. To so, run::
tutor images build openedx-dev
Alternatively, the image will be automatically rebuilt every time you run::
tutor dev launch
.._bind_mounts:
Bind-mount container directories
--------------------------------
Sharing directories with containers
-----------------------------------
It may sometimes be convenient to mount container directories on the host, for instance: for editing and debugging. Tutor provides different solutions to this problem.
Bind-mount from the "volumes/" directory
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.._persistent_mounts:
Tutor makes it easy to create a bind-mount from an existing container. First, copy the contents of a container directory with the ``bindmount`` command. For instance, to copy the virtual environment of the "lms" container::
Persistent bind-mounted volumes with ``tutor mounts``
``tutor mounts`` is a set of Tutor command to manage bind-mounted host directories. Directories are mounted `both` at build time and run time:
This command recursively copies the contents of the ``/opendedx/venv`` directory to ``$(tutor config printroot)/volumes/venv``. The code of any Python dependency can then be edited -- for instance, you can then add a ``import ipdb; ipdb.set_trace()`` statement for step-by-step debugging, or implement a custom feature.
- At build time: some of the host directories will be added the `Docker build context <https://docs.docker.com/engine/reference/commandline/buildx_build/#build-context>`__. This makes it possible to transparently build a Docker image using a locally checked-out repository.
- At run time: host directories will be bind-mounted in running containers, using either an automatic or a manual configuration.
Then, bind-mount the directory back in the container with the ``--volume`` option::
tutor dev runserver --volume=/openedx/venv lms
After some directories have been added with ``tutor mounts add``, all ``tutor dev`` and ``tutor local`` commands will make use of these bind-mount volumes.
Notice how the ``--volume=/openedx/venv`` option differs from `Docker syntax <https://docs.docker.com/storage/volumes/#choose-the--v-or---mount-flag>`__? Tutor recognizes this syntax and automatically converts this option to ``--volume=/path/to/tutor/root/volumes/venv:/openedx/venv``, which is recognized by Docker.
Values passed to ``tutor mounts add ...`` can take one of two forms. The first is explicit::
..note::
The ``bindmount`` command and the ``--volume=/...`` option syntax are available both for the ``tutor local`` and ``tutor dev`` commands.
With the explicit form, the value means "bind-mount the host folder /path/to/edx-platform to /openedx/edx-platform in the lms container at run time".
If you use the explicit format, you will quickly realise that you usually want to bind-mount folders in multiple containers at a time. For instance, you will want to bind-mount the edx-platform repository in the "cms" container, but also the "lms-worker" and "cms-worker" containers. To do that, write instead::
This command line is a bit cumbersome. In addition, with this explicit form, the edx-platform repository will *not* be added to the build context at build time. But Tutor can be smart about bind-mounting folders to the right containers in the right place when you use the implicit form of the ``tutor mounts add`` command. For instance, the following implicit form can be used instead of the explicit form above::
tutor mounts add /path/to/edx-platform
With this implicit form, the edx-platform repo will be bind-mounted in the containers at run time, just like with the explicit form. But in addition, the edx-platform will also automatically be added to the Docker image at build time.
To check whether you have used the correct syntax, you should run ``tutor mounts list``. This command will indicate whether your folders will be bind-mounted at build time, run time, or both. For instance::
$ tutor mounts add /path/to/edx-platform
$ tutor mounts list
- name: /path/to/edx-platform
build_mounts:
- image: openedx
context: edx-platform
- image: openedx-dev
context: edx-platform
compose_mounts:
- service: lms
container_path: /openedx/edx-platform
- service: cms
container_path: /openedx/edx-platform
- service: lms-worker
container_path: /openedx/edx-platform
- service: cms-worker
container_path: /openedx/edx-platform
- service: lms-job
container_path: /openedx/edx-platform
- service: cms-job
container_path: /openedx/edx-platform
So, when should you *not* be using the implicit form? That would be when Tutor does not know where to bind-mount your host folders. For instance, if you wanted to bind-mount your edx-platform virtual environment located in ``~/venvs/edx-platform``, you should not write ``mounts add ~/venvs/edx-platform``, because that folder would be mounted in a way that would override the edx-platform repository in the container. Instead, you should write::
Sometimes, you may want to modify some of the files inside a container for which you don't have a copy on the host. A typical example is when you want to troubleshoot a Python dependency that is installed inside the application virtual environment. In such cases, you want to first copy the contents of the virtual environment from the container to the local filesystem. To that end, Tutor provides the ``tutor dev copyfrom`` command. First, copy the contents of the container folder to the local filesystem::
tutor dev copyfrom lms /openedx/venv ~
Then, bind-mount that folder back in the container with the ``MOUNTS`` setting (described :ref:`above <persistent_mounts>`)::
tutor mounts add lms:~/venv:/openedx/venv
You can then edit the files in ``~/venv`` on your local filesystem and see the changes live in your "lms" container.
Manual bind-mount to any directory
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The above solution may not work for you if you already have an existing directory, outside of the "volumes/" directory, which you would like mounted in one of your containers. For instance, you may want to mount your copy of the `edx-platform <https://github.com/edx/edx-platform/>`__ repository. In such cases, you can simply use the ``-v/--volume```Docker option <https://docs.docker.com/storage/volumes/#choose-the--v-or---mount-flag>`__::
..warning:: Manually bind-mounting volumes with the ``--volume`` option makes it difficult to simultaneously bind-mount to multiple containers. Also, the ``--volume`` options are not compatible with ``start`` commands. As an alternative, you should consider following the instructions above: :ref:`persistent_mounts`.
The above solution may not work for you if you already have an existing directory, outside of the "volumes/" directory, which you would like mounted in one of your containers. For instance, you may want to mount your copy of the `edx-platform <https://github.com/openedx/edx-platform/>`__ repository. In such cases, you can simply use the ``-v/--volume```Docker option <https://docs.docker.com/storage/volumes/#choose-the--v-or---mount-flag>`__::
tutor dev run --volume=/path/to/edx-platform:/openedx/edx-platform lms bash
Override docker-compose volumes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The above solutions require that you explicitly pass the ``-v/--volume`` to every ``run`` or ``runserver`` command, which may be inconvenient. Also, these solutions are not compatible with the ``start`` command. To address these issues, you can create a ``docker-compose.override.yml`` file that will specify custom volumes to be used with all ``dev`` commands::
..warning:: While the option described below "works", it will only bind-mount directories at run-time. In many cases you really want to bind-mount directories at build-time. For instance: when working on edx-platform requirements. As an alternative, you should consider following the instructions above: :ref:`persistent_mounts`.
Adding items to the ``MOUNTS`` setting effectively adds new bind-mount volumes to the ``docker-compose.yml`` files. But you might want to have more control over your volumes, such as adding read-only options, or customising other fields of the different services. To address these issues, you can create a ``docker-compose.override.yml`` file that will specify custom volumes to be used with all ``dev`` commands::
vim "$(tutor config printroot)/env/dev/docker-compose.override.yml"
@ -101,170 +242,18 @@ You are then free to bind-mount any directory to any container. For instance, to
services:
lms:
volumes:
- /path/to/edx-platform/:/openedx/edx-platform
- /path/to/edx-platform:/openedx/edx-platform
cms:
volumes:
- /path/to/edx-platform/:/openedx/edx-platform
- /path/to/edx-platform:/openedx/edx-platform
lms-worker:
volumes:
- /path/to/edx-platform/:/openedx/edx-platform
- /path/to/edx-platform:/openedx/edx-platform
cms-worker:
volumes:
- /path/to/edx-platform/:/openedx/edx-platform
- /path/to/edx-platform:/openedx/edx-platform
This override file will be loaded when running any ``tutor dev ..`` command. The edx-platform repo mounted at the specified path will be automatically mounted inside all LMS and CMS containers. With this file, you should no longer specify the ``-v/--volume`` option from the command line with the ``run`` or ``runserver`` commands.
This override file will be loaded when running any ``tutor dev ..`` command. The edx-platform repo mounted at the specified path will be automatically mounted inside all LMS and CMS containers.
..note::
The ``tutor local`` commands loads the ``docker-compose.override.yml`` file from the ``$(tutor config printroot)/env/local/docker-compose.override.yml`` directory.
Point to a local edx-platform
-----------------------------
Following the instructions :ref:`above <bind_mounts>` on how to bind-mount directories from the host above, you may mount your own `edx-platform <https://github.com/edx/edx-platform/>`__ fork in your containers by running either::
# Mount from the volumes/ directory
tutor dev bindmount lms /openedx/edx-platform
tutor dev runserver --volume=/openedx/edx-platform lms
# Mount from an arbitrary directory
tutor dev runserver --volume=/path/to/edx-platform:/openedx/edx-platform lms
# Add your own volumes to $(tutor config printroot)/env/dev/docker-compose.override.yml
tutor dev runserver lms
Prepare the edx-platform repo
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
If you choose any but the first solution above, you will have to make sure that your fork works with Tutor.
First of all, you should make sure that you are working off the ``open-release/lilac.1`` tag. See the :ref:`fork edx-platform section <edx_platform_fork>` for more information.
To debug a local edx-platform repository, add a ``import ipdb; ipdb.set_trace()`` breakpoint anywhere in your code and run::
tutor dev runserver [--volume=...] lms
XBlock and edx-platform plugin development
------------------------------------------
In some cases you will have to develop features for packages that are pip-installed next to edx-platform. This is quite easy with Tutor. Just add your packages to the ``$(tutor config printroot)/env/build/openedx/requirements/private.txt`` file. To avoid re-building the openedx Docker image at every change, you should add your package in editable mode. For instance::
The ``requirements`` folder should have the following content::
env/build/openedx/requirements/
private.txt
mypackage/
setup.py
...
You will have to re-build the openedx Docker image once::
tutor images build openedx
You should then run the development server as usual, with ``runserver``. Every change made to the ``mypackage`` folder will be picked up and the development server will be automatically reloaded.
.._theming:
Customised themes
-----------------
With Tutor, it's pretty easy to develop your own themes. Start by placing your files inside the ``env/build/openedx/themes`` directory. For instance, you could start from the ``edx.org`` theme present inside the ``edx-platform`` repository::
You should not create a soft link here. If you do, it will trigger a ``Theme not found in any of the themes dirs`` error. This is because soft links are not properly resolved from inside docker containers.
Then, run a local webserver::
tutor dev runserver lms
The LMS can then be accessed at http://local.overhang.io:8000. You will then have to :ref:`enable that theme <settheme>` for the development domain names::
tutor dev settheme mythemename local.overhang.io:8000 studio.local.overhang.io:8001
Re-build development docker image (and compile assets)::
tutor images build openedx-dev
Watch the themes folders for changes (in a different terminal)::
tutor dev run watchthemes
Make changes to some of the files inside the theme directory: the theme assets should be automatically recompiled and visible at http://local.overhang.io:8000.
Custom edx-platform settings
----------------------------
By default, tutor settings files are mounted inside the docker images at ``/openedx/edx-platform/lms/envs/tutor/`` and ``/openedx/edx-platform/cms/envs/tutor/``. In the various ``dev`` commands, the default ``edx-platform`` settings module is set to ``tutor.development`` and you don't have to do anything to set up these settings.
If, for some reason, you want to use different settings, you will need to define the ``TUTOR_EDX_PLATFORM_SETTINGS`` environment variable.
For instance, let's assume you have created the ``/path/to/edx-platform/lms/envs/mysettings.py`` and ``/path/to/edx-platform/cms/envs/mysettings.py`` modules. These settings should be pretty similar to the following files::
Alternatively, the ``mysettings.py`` files can import the tutor development settings::
# Beginning of mysettings.py
from .tutor.development import *
You should then specify the settings to use on the host::
export TUTOR_EDX_PLATFORM_SETTINGS=mysettings
From then on, all ``dev`` commands will use the ``mysettings`` module. For instance::
tutor dev runserver lms
Running edx-platform unit tests
-------------------------------
It's possible to run the full set of unit tests that ship with `edx-platform <https://github.com/edx/edx-platform/>`__. To do so, you should first build the "test" target of the "openedx-dev" Docker image::
tutor images build --target=test openedx-dev
..warning::
Don't forget to re-build the development image afterwards if you'd like to run ``dev`` commands again! To do so, run ``tutor images build openedx-dev`` after you are done testing.
Then, run unit tests with ``pytest`` commands::
# Run a test container
tutor dev run lms bash
# Run tests on common apps
unset DJANGO_SETTINGS_MODULE
export EDXAPP_TEST_MONGO_HOST=mongodb
pytest common
pytest openedx
# Run tests on LMS
export DJANGO_SETTINGS_MODULE=lms.envs.tutor.test
pytest lms
# Run tests on CMS
export DJANGO_SETTINGS_MODULE=cms.envs.tutor.test
pytest cms
..note::
Getting all edx-platform unit tests to pass on Tutor is currently a work-in-progress. Some unit tests are still failing. If you manage to fix some of these, please report your findings in the `Tutor forums <https://discuss.overhang.io>`__.
The ``tutor local`` commands load the ``docker-compose.override.yml`` file from the ``$(tutor config printroot)/env/local/docker-compose.override.yml`` directory. One-time jobs from initialisation commands load the ``local/docker-compose.jobs.override.yml`` and ``dev/docker-compose.jobs.override.yml``.
Tutor is an open source distribution of `Open edX <https://open.edx.org>`_. It uses the original code from the various Open edX repositories, such as `edx-platform <https://github.com/edx/edx-platform/>`_, `cs_comments_service <https://github.com/edx/cs_comments_service>`_, etc. and packages everything in a way that makes it very easy to install, administer and upgrade Open edX. In particular, all services are run inside Docker containers.
Tutor is an open source distribution of `Open edX <https://open.edx.org>`_. It uses the original code from the various Open edX repositories, such as `edx-platform <https://github.com/openedx/edx-platform/>`_, `cs_comments_service <https://github.com/openedx/cs_comments_service>`_, etc. and packages everything in a way that makes it very easy to install, administer and upgrade Open edX. In particular, all services are run inside Docker containers.
Tutor makes it possible to deploy Open edX locally, with `docker-compose <https://docs.docker.com/compose/overview/>`_ or on an existing `Kubernetes cluster <http://kubernetes.io/>`_.
Tutor makes it possible to deploy Open edX locally, with `docker-compose <https://docs.docker.com/compose/overview/>`_ or on an existing `Kubernetes cluster <http://kubernetes.io/>`_. Want to learn more? Take a look at the :ref:`getting started concepts <intro>`.
What is the purpose of Tutor?
-----------------------------
@ -20,44 +20,39 @@ To make it possible to deploy, administer and upgrade Open edX anywhere, easily.
What's the difference with the official "native" installation?
The `native installation <https://openedx.atlassian.net/wiki/spaces/OpenOPS/pages/146440579/Native+Open+edX+Ubuntu+16.04+64+bit+Installation>`_ maintained by edX relies on `Ansible scripts <https://github.com/edx/configuration/>`_ to deploy Open edX on one or multiple servers. These scripts suffer from a couple issues that Tutor tries to address:
The `native installation <https://openedx.atlassian.net/wiki/spaces/OpenOPS/pages/146440579/Native+Open+edX+Ubuntu+16.04+64+bit+Installation>`_ maintained by edX relies on `Ansible scripts <https://github.com/openedx/configuration/>`_ to deploy Open edX on one or multiple servers. These scripts suffer from a couple of issues that Tutor tries to address:
1. Complexity: the scripts contain close to 35k lines of code spread over 780 files. They are really hard to understand, debug, and modify, and they are extremly slow. As a consequence, Open edX is often wrongly perceived as a project that is overly complex to manage. In contrast, Tutor generates mostly ``Dockerfile`` and ``docker-compose.yml`` files that make it easy to understand what is going on. Also, the whole installation should take about 10 minutes.
1. Complexity: the scripts contain close to 35k lines of code spread over 780 files. They are really hard to understand, debug, and modify, and they are extremely slow. As a consequence, Open edX is often wrongly perceived as a project that is overly complex to manage. In contrast, Tutor generates mostly ``Dockerfile`` and ``docker-compose.yml`` files that make it easy to understand what is going on. Also, the whole installation should take about 10 minutes.
2. Isolation from the OS: Tutor barely needs to touch your server because the entire platform is packaged inside Docker containers. You are thus free to run other services on your server without fear of indirectly crashing your Open edX platform.
3. Compatibility: Open edX is only compatible with Ubuntu 16.04, but that shouldn't mean you are forced to run this specific OS. With Tutor, you can deploy Open edX on just any server you like: Ubuntu 18.04, Red Hat, Debian... All docker-compatible platforms are supported.
4. Security: because you are no longer bound to a single OS, with Tutor you are now free to install security-related upgrades as soon as they become available.
5. Portability: Tutor makes it easy to move your platform from one server to another. Just zip-compress your Tutor project root, send it to another server and you're done.
There are also many features that are not included in the native installation, such as a `web user interface <https://github.com/overhangio/tutor-webui>`__ for remotely installing the platform, :ref:`Kubernetes deployment <k8s>`, additional languages, etc. You'll discover these differences as you explore Tutor :)
Many features that are not included in the native installation, such as a `web user interface <https://github.com/overhangio/tutor-webui>`__ for remotely installing the platform, :ref:`Kubernetes deployment <k8s>`, additional languages, etc. You'll discover these differences as you explore Tutor :)
What's the difference with the official devstack?
-------------------------------------------------
The `devstack <https://github.com/edx/devstack>`_ is meant for development only, not for production deployment. Tutor can be used both for production deployment and :ref:`locally hacking on Open edX <development>`.
The `devstack <https://github.com/openedx/devstack>`_ is meant for development only, not for production deployment. Tutor can be used both for production deployment and :ref:`locally hacking on Open edX <development>`.
Is Tutor officially supported by edX?
-------------------------------------
No. Tutor is developed independently from edX. That means that the folks at edX.org are *not* responsible for troubleshooting issues of this project. Please don't bother Ned ;-)
Yes: as of the Open edX Maple release (December 9th 2021), Tutor is the only officially supported installation method for Open edX: see the `official installation instructions <https://edx.readthedocs.io/projects/edx-installing-configuring-and-running/en/open-release-quince.master/installation/index.html>`__.
What features are missing from Tutor?
-------------------------------------
Tutor tries very hard to support all major Open edX features, notably in the form of :ref:`plugins <existing_plugins>`. In particular, the discovery and ecommerce services, once unavailable in Tutor, can now be easily installed via plugins. If you are interested in sponsoring the development of a new plugin, please `get in touch <mailto:worktogether@overhang.io>`__!
Tutor tries very hard to support all major Open edX features, notably in the form of :ref:`plugins <existing_plugins>`. If you are interested in sponsoring the development of a new plugin, please `get in touch <mailto:worktogether@overhang.io>`__!
It should be noted that the `Analytics <https://github.com/edx/edx-analytics-pipeline>`__ stack is currently unsupported, and will likely stay so in the future, as it would require a tremendous amount of work to containerize all the components. We are currently working on a replacement solution.
It should be noted that the `Insights <https://github.com/openedx/edx-analytics-pipeline>`__ stack is currently unsupported, because of its complexity, lack of support, and extensibility. To replace it, we developed `Cairn <https://github.com/overhangio/tutor-cairn>`__ the next-generation analytics solution for Open edX. You should check it out 😉
Are there people already running this in production?
Yes, many of them. There is no way to count precisely how many running Open edX platforms were deployed with Tutor, but from feedback collected directly from real users, there must be hundreds, if not thousands. Tutor is also used by some Open edX providers who are hosting platforms for their customers.
Yes: system administrators all around the world use Tutor to run their Open edX platforms, from single-class school teachers to renowned universities, Open edX SaaS providers, and nation-wide learning platforms.
Why should I trust software written by some random guy on the Internet?
You shouldn't :) Tutor is actively maintained by `Overhang.IO <https://overhang.io>`_, a France-based company founded by `Régis Behmo <https://github.com/regisb/>`_. Régis has been working on Tutor since early 2018; he has been a contributor of the Open edX project since 2015. In particular, he has worked for 2 years at `FUN-MOOC <https://www.fun-mooc.fr/>`_, one of the top 5 largest Open edX platforms in the world. He presented several talks at the Open edX conferences:
- *Deploying a robust, scalable Open edX platform in 1 click (or less) with Tutor*, March 2019 (`video <https://www.youtube.com/watch?v=Oqc7c-3qFc4>`_, `slides <https://regisb.github.io/openedx2019/>`_)
- *Videofront: a Self-Hosted YouTube*, June 2017 (`video <https://www.youtube.com/watch?v=e7bJchJrmP8&t=5m53s>`__, `slides <http://regisb.github.io/openedx-conference-2017/>`__)
- *Open edX 101: A Source Code Review*, June 2016 (`video <https://www.youtube.com/watch?v=DVku7Y7XQII>`__, `slides <http://regisb.github.io/openedx-conference-2016/>`__)
- *FUN: Life in the Avant-Garde*, Oct. 2015 (`video <https://www.youtube.com/watch?v=V1EBo1l8BgY>`__, `slides <http://regisb.github.io/openedx-conference-2015/>`__)
You shouldn't :) Tutor is actively maintained by `Edly <https://edly.io>`__, a US-based ed-tech company facilitating over 40 million learners worldwide through its eLearning solutions. With a credible engineering team that has won clients' hearts globally led by `Régis Behmo <https://github.com/regisb/>`__, Tutor has empowered numerous edtech ventures over the years. Additionally, Tutor is a `community-led project <https://github.com/overhangio/tutor>`__ with many contributions from its :ref:`project maintainers <maintainers>`.
@ -47,6 +62,6 @@ This work is licensed under the terms of the `GNU Affero General Public License
The AGPL license covers the Tutor code, including the Dockerfiles, but not the content of the Docker images which can be downloaded from https://hub.docker.com. Software other than Tutor provided with the docker images retain their original license.
The Tutor plugin system is licensed under the terms of the `Apache License, Version 2.0 <https://opensource.org/licenses/Apache-2.0>`__.
The Tutor plugin and hooks system is licensed under the terms of the `Apache License, Version 2.0 <https://opensource.org/licenses/Apache-2.0>`__.
* Supported OS: Tutor runs on any 64-bit, UNIX-based system. It was also reported to work on Windows.
* Supported OS: Tutor runs on any 64-bit, UNIX-based OS. It was also reported to work on Windows (with `WSL 2 <https://docs.microsoft.com/en-us/windows/wsl/install>`__).
* Architecture: Both AMD64 and ARM64 are supported.
Do not attempt to simply run ``apt-get install docker docker-compose`` on older Ubuntu platforms, such as 16.04 (Xenial), as you will get older versions of these utilities.
* Ports 80 and 443 should be open. If other web services run on these ports, check the section on :ref:`how to setup a web proxy <web_proxy>`.
* Ports 80 and 443 should be open. If other web services run on these ports, check the tutorial on :ref:`how to setup a web proxy <web_proxy>`.
* Hardware:
- Minimum configuration: 4 GB RAM, 2 CPU, 8 GB disk space
@ -26,56 +27,56 @@ Requirements
..note::
On Mac OS, by default, containers are allocated 2 GB of RAM, which is not enough. You should follow `these instructions from the official Docker documentation <https://docs.docker.com/docker-for-mac/#advanced>`__ to allocate at least 4-5 GB to the Docker daemon. If the deployment fails because of insufficient memory during database migrations, check the :ref:`relevant section in the troubleshooting guide <migrations_killed>`.
.._install_binary:
Download
--------
Direct binary download
----------------------
Choose **one** of the installation methods below. If you install Tutor in different ways, you will end up with multiple ``tutor`` executables, which is going to be very confusing. At any time, you can check the path to your ``tutor`` executable by running ``which tutor``.
The latest binaries can be downloaded from https://github.com/overhangio/tutor/releases. From the command line:
Python package
~~~~~~~~~~~~~~
..include::cli_download.rst
..include:: download/pip.rst
This is the simplest and recommended installation method for most people. Note however that you will not be able to use custom plugins with this pre-compiled binary. The only plugins you can use with this approach are those that are already bundled with the binary: see the :ref:`existing plugins <existing_plugins>`.
.._install_source:
Alternative installation methods
--------------------------------
If you would like to inspect the Tutor source code, you are most welcome to install Tutor from `Pypi <https://pypi.org/project/tutor-openedx/>`_ or directly from `the Github repository <https://github.com/overhangio/tutor>`_. You will need python >= 3.6 with pip and the libyaml development headers. On Ubuntu, these requirements can be installed by running::
Check the "tutor" package on Pypi: https://pypi.org/project/tutor. You will need Python >= 3.6 with pip and the libyaml development headers. On Ubuntu, these requirements can be installed by running::
sudo apt install python3 python3-pip libyaml-dev
Installing from pypi
~~~~~~~~~~~~~~~~~~~~
.._install_binary:
::
Binary release
~~~~~~~~~~~~~~
pip install tutor-openedx
The latest binaries can be downloaded from https://github.com/overhangio/tutor/releases. From the command line:
..include:: download/binary.rst
This is the simplest and recommended installation method for most people who do not have Python 3 on their machine. Note however that **you will not be able to use custom plugins** with this pre-compiled binary. The only plugins you can use with this approach are those that are already bundled with the binary: see the :ref:`existing plugins <existing_plugins>`.
.._install_source:
Installing from source
~~~~~~~~~~~~~~~~~~~~~~
::
To inspect the Tutor source code, install Tutor from `the Github repository <https://github.com/overhangio/tutor>`__::
git clone https://github.com/overhangio/tutor
cd tutor
pip install -e .
DNS records
-----------
Configuring DNS records
-----------------------
When running a server in production, it is necessary to define `DNS records <https://en.wikipedia.org/wiki/Domain_Name_System#Resource_records>`__ which will make it possible to access your Open edX platform by name in your browser. The precise procedure to create DNS records vary from one provider to the next and is beyond the scope of these docs. You should create a record of type A with a name equal to your LMS hostname (given by ``tutor config printvalue LMS_HOST``) and a value that indicates the IP address of your server. Applications other than the LMS, such as the studio, ecommerce, etc. typically reside in subdomains of the LMS. Thus, you should also create a CNAME record to point all subdomains of the LMS to the LMS_HOST.
When running a server in production, it is necessary to define `DNS records <https://en.wikipedia.org/wiki/Domain_Name_System#Resource_records>`__ which will make it possible to access your Open edX platform by name in your browser. The precise procedure to create DNS records varies from one provider to the next and is beyond the scope of these docs. You should create a record of type A with a name equal to your LMS hostname (given by ``tutor config printvalue LMS_HOST``) and a value that indicates the IP address of your server. Applications other than the LMS, such as the studio, ecommerce, etc. typically reside in subdomains of the LMS. Thus, you should also create a CNAME record to point all subdomains of the LMS to the LMS_HOST.
For instance, the demo Open edX server that runs at http://demo.openedx.overhang.io has the following DNS records::
For instance, to run an Open edX server at https://learn.mydomain.com on a server with IP address 1.1.1.1, you would need to configure the following DNS records::
demo.openedx 1800 IN A 172.105.89.208
*.demo.openedx 1800 IN CNAME demo.openedx.overhang.io.
learn 1800 IN A 1.1.1.1
*.learn 1800 IN CNAME learn.mydomain.com.
.._cloud_install:
Zero-click AWS installation
~~~~~~~~~~~~~~~~~~~~~~~~~~~
---------------------------
Tutor can be launched on Amazon Web Services very quickly with the `official Tutor AMI <https://aws.amazon.com/marketplace/pp/B07PV3TB8X>`__. Shell access is not required, as all configuration will happen through the Tutor web user interface. For detailed installation instructions, we recommend watching the following video:
@ -86,42 +87,77 @@ Tutor can be launched on Amazon Web Services very quickly with the `official Tut
Upgrading
---------
With Tutor, it is very easy to upgrade to a more recent Open edX or Tutor release. Just install the latest ``tutor`` version (using either methods above) and run the ``quickstart`` command again. If you have :ref:`customised <configuration_customisation>` your docker images, you will have to re-build them prior to running ``quickstart``.
To upgrade your Open edX site or benefit from the latest features and bug fixes, you should simply upgrade Tutor. Start by backing up your data and reading the `release notes <https://docs.openedx.org/en/latest/community/release_notes/>`_ for the current release.
``quickstart`` should take care of automatically running the upgrade process. If for some reason you need to *manually* upgrade from an Open edX release to the next, you should run ``tutor local upgrade``. For instance, to upgrade from Koa to Lilac, run::
Next, upgrade the "tutor" package and its dependencies::
tutor local upgrade --from=koa
pip install --upgrade "tutor[full]"
Then run the ``launch`` command again. Depending on your deployment target, run one of::
tutor local launch # for local installations
tutor dev launch # for local development installations
tutor k8s launch # for Kubernetes installation
Upgrading with custom Docker images
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
If you run :ref:`customised <configuration_customisation>` Docker images, you need to rebuild them before running ``launch``::
tutor config save
tutor images build all # specify here the images that you need to build
tutor local launch
Upgrading to a new Open edX release
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Major Open edX releases are published twice a year, in June and December, by the Open edX `Build/Test/Release working group <https://discuss.openedx.org/c/working-groups/build-test-release/30>`__. When a new Open edX release comes out, Tutor gets a major version bump (see :ref:`versioning`). Such an upgrade typically includes multiple breaking changes. Any upgrade is final because downgrading is not supported. Thus, when upgrading your platform from one major version to the next, it is strongly recommended to do the following:
1. Read the changes listed in the `CHANGELOG.md <https://github.com/overhangio/tutor/blob/master/CHANGELOG.md>`__ file. Breaking changes are identified by a "💥".
2. Perform a backup (see the :ref:`backup tutorial <backup_tutorial>`). On a local installation, this is typically done with::
3. If you created custom plugins, make sure that they are compatible with the newer release.
4. Test the new release in a sandboxed environment.
5. If you are running edx-platform, or some other repository from a custom branch, then you should rebase (and test) your changes on top of the latest release tag (see :ref:`edx_platform_fork`).
The process for upgrading from one major release to the next works similarly to any other upgrade, with the ``launch`` command (see above). The single difference is that if the ``launch`` command detects that your tutor environment was generated with an older release, it will perform a few release-specific upgrade steps. These extra upgrade steps will be performed just once. But they will be ignored if you updated your local environment (for instance: with ``tutor config save``) before running ``launch``. This situation typically occurs if you need to re-build some Docker images (see above). In such a case, you should make use of the ``upgrade`` command. For instance, to upgrade a local installation from Palm to Quince and rebuild some Docker images, run::
tutor config save
tutor images build all # list the images that should be rebuilt here
tutor local upgrade --from=palm
tutor local launch
.._autocomplete:
Autocomplete
------------
Shell autocompletion
--------------------
Tutor is built on top of `Click <https://click.palletsprojects.com>`_, which is a great library for building command line interface (CLI) tools. As such, Tutor benefits from all Click features, including `auto-completion <https://click.palletsprojects.com/en/7.x/bashcomplete/>`_. After installing Tutor, auto-completion can be enabled by running::
Tutor is built on top of `Click <https://click.palletsprojects.com>`_, which is a great library for building command line interface (CLI) tools. As such, Tutor benefits from all Click features, including `auto-completion <https://click.palletsprojects.com/en/8.x/bashcomplete/>`_. After installing Tutor, auto-completion can be enabled in bash by running::
_TUTOR_COMPLETE=source tutor >> ~/.bashrc
_TUTOR_COMPLETE=bash_source tutor >> ~/.bashrc
If you are running zsh, run instead::
_TUTOR_COMPLETE=source_zsh tutor >> ~/.zshrc
_TUTOR_COMPLETE=zsh_source tutor >> ~/.zshrc
After opening a new shell, you can test auto-completion by typing::
tutor <tab><tab>
..include:: podman.rst
Uninstallation
--------------
It is fairly easy to completely uninstall Tutor and to delete the Open edX platforms that is running locally.
It is fairly easy to completely uninstall Tutor and to delete the Open edX platforms that are running locally.
First of all, stop any locally-running platform::
First of all, stop any locally-running platform and remove all Tutor containers::
tutor local stop
tutor dev stop
tutor local dc down --remove-orphans
tutor dev dc down --remove-orphans
Then, delete all data associated to your Open edX platform::
Then, delete all data associated with your Open edX platform::
# WARNING: this step is irreversible
sudo rm -rf "$(tutor config printroot)"
@ -129,7 +165,13 @@ Then, delete all data associated to your Open edX platform::
Finally, uninstall Tutor itself::
# If you installed tutor from source
pip uninstall tutor-openedx
pip uninstall tutor
# If you downloaded the tutor binary
sudo rm /usr/local/bin/tutor
# Optionally, you may want to remove Tutor plugins installed.
# You can get a list of the installed plugins:
pip freeze | grep tutor
# You can then remove them using the following command:
`Open edX <http://open.edx.org/>`_ is a thriving open source project, backed by a great community, for running an online learning platform at scale. Open edX comes with an LMS (Learning Management System) where students access course contents, a CMS (Content Management System) that course staff uses to design courses, and a few other components to provide more services to students, course staff and platform administrators.
`Open edX <http://open.edx.org/>`_ is a thriving open source project, backed by a great community, for running an online learning platform at scale. Open edX comes with an LMS (Learning Management System) where students access course contents, a CMS (Content Management System) that course staff uses to design courses, and a few other components to provide more services to students, course staff, and platform administrators.
Should I use Open edX?
----------------------
@ -18,77 +18,118 @@ Open edX competitors include `Moodle <https://moodle.org/>`__, `Instructure's Ca
* Multiple extension points for comprehensive customization
* Modern, intuitive user interface to keep students engaged
Open edX is a safe bet: it is backed by edX.org, a US-based non-profit that is committed to open source and which runs Open edX to service its millions of learners. With Open edX you can be sure that the features you need will be available. If it's good enough for Harvard, the MIT or the French government, then it will probably also work for you.
Open edX is a safe bet: it is backed by edX.org, a US-based non-profit that is committed to open source and which runs Open edX to service its millions of learners. With Open edX you can be sure that the features you need will be available. If it's good enough for Harvard, the MIT, or the French government, then it will probably also work for you.
Should I self-host Open edX or rely on a hosting provider?
Third-party Open edX providers can provide you with custom, closed-source features that they developed internally. However, their pricing is usually per-seat: that makes it difficult to predict how much running Open edX will actually cost you if you don't know in advance how many students will connect to your platform. And once you start scaling up and adding many students, running the platform will become very expensive.
On the other hand, running Open edX on your own servers will help you keep your costs under control. Because you own your servers and data, you will always be able to migrate your platform, either to a different cloud provider or an Open edX service provider. This is the true power of open source.
On the other hand, running Open edX on your own servers will help you keep your costs under control. Because you own your servers and data, you will always be able to migrate your platform, either to a different cloud provider or an Open edX service provider. This is the true power of the open source.
Should I use Tutor?
-------------------
Running software onpremises is cheaper only if your management costs don't go through the roof. You do not want to hire a full-time devops team just for managing your online learning platform. This is why we created Tutor: to make it easy to run a state-of-the-art online learning platform without breaking the bank. Historically, it's always been difficult to install Open edX with the native installation scripts. For instance, there are no official instructions for upgrading an existing Open edX platform: the `recommended approach <https://docs.bitnami.com/azure/apps/edx/administration/upgrade/>`__ is to backup all data, trash the server and create a new one. As a consequence, people tend not to upgrade their platform and keep running on deprecated releases. Tutor makes it possible even to non-technical users to launch, manage and upgrade Open edX at any scale. Should you choose at some point that Tutor is not the right solution for you, you always have an escape route: because Tutor is open source software, you can easily dump your data and switch to a different installation method. But we are confident you will not do that 😉
Running software on-premises is cheaper only if your management costs don't go through the roof. You do not want to hire a full-time devops team just for managing your online learning platform. This is why we created Tutor: to make it easy to run a state-of-the-art online learning platform without breaking the bank. Historically, it's always been difficult to install Open edX with native installation scripts. For instance, there are no official instructions for upgrading an existing Open edX platform: the `recommended approach <https://docs.bitnami.com/azure/apps/edx/administration/upgrade/>`__ is to backup all data, trash the server, and create a new one. As a consequence, people tend not to upgrade their platform and keep running on deprecated releases. Tutor makes it possible even for non-technical users to launch, manage and upgrade Open edX at any scale. Should you choose at some point that Tutor is not the right solution for you, you always have an escape route: because Tutor is open source software, you can easily dump your data and switch to a different installation method. But we are confident you will not do that 😉
To learn more about Tutor, watch this 7-minute lightning talk that was made at the 2019 Open edX conference in San Diego, CA (with `slides <https://regisb.github.io/openedx2019/>`_):
..youtube:: Oqc7c-3qFc4
How does Tutor work, technically speaking?
------------------------------------------
How does Tutor simplify Open edX deployment?
--------------------------------------------
Tutor simplifies the deployment of Open edX by:
1. Separating the configuration logic from the deployment platforms.
2. Running application processes in cleanly separated `docker containers <https://www.docker.com/resources/what-container>`_.
3. Providing user-friendly, reliable commands for common administration tasks, including upgrades and monitoring.
4. Using a simple :ref:`plugin system <plugins>` that makes it easy to extend and customize Open edX.
4. Using a simple :ref:`plugin system <plugins>` that makes it easy to extend and customise Open edX.
Because Docker containers are becoming an industry-wide standard, that means that with Tutor it becomes possible to run Open edX anywhere: for now, Tutor supports deploying on a local server, with `docker-compose <https://docs.docker.com/compose/overview/>`_, and in a large cluster, with `Kubernetes <http://kubernetes.io/>`_. But in the future, Tutor may support other deployment platforms.
Because Docker containers are becoming an industry-wide standard, that means that with Tutor it becomes possible to run Open edX anywhere: for now, Tutor supports deploying on a local server, with `dockercompose <https://docs.docker.com/compose/overview/>`_, and in a large cluster, with `Kubernetes <http://kubernetes.io/>`_. But in the future, Tutor may support other deployment platforms.
Where can I try Open edX and Tutor?
-----------------------------------
A demo Open edX platform is available at https://demo.openedx.overhang.io. This platform was deployed using Tutor and the `Indigo theme <https://github.com/overhangio/indigo>`__. Feel free to play around with the following credentials:
A sandbox Open edX platform is available at https://sandbox.openedx.edly.io. This platform was deployed using Tutor and the `Indigo theme <https://github.com/overhangio/indigo>`__. Feel free to play around with the following credentials:
The Android mobile application for this website can be downloaded at this url: http://demo.openedx.overhang.io/static/mobile/app.apk
The Android mobile application for this demo platform can be downloaded at this url: https://mobile.sandbox.openedx.edly.io/app.apk
Urls:
* LMS: https://demo.openedx.overhang.io
* Studio (CMS): https://studio.demo.openedx.overhang.io
* LMS: https://sandbox.openedx.edly.io
* Studio (CMS): https://studio.sandbox.openedx.edly.io
The platform is reset every day at 9:00 AM, `Paris (France) time <https://time.is/Paris>`__, so feel free to try and break things as much as you want.
.._how_does_tutor_work:
How does Tutor work?
--------------------
Tutor is a piece of software that takes care of exactly three things:
1. Project configuration: user-specific settings (such as secrets) are stored in a single ``config.yml`` file.
2. Template rendering: all the files that are necessary to run your platform are generated from a set of templates and user-specific settings.
3. Command-line interface (CLI): frequently-used administration commands are gathered in a convenient, unified CLI.
You can experiment with Tutor very quickly: start by `installing <install>`_ Tutor. Then run::
tutor config save --interactive
$ tutor config save --interactive
This command does two things:
Then, to view the result of the above command::
1. Generate a ``config.yml`` configuration file: this file contains core :ref:`configuration parameters <configuration>` for your Open edX platforms, such as passwords and feature flags.
2. Generate an ``env/`` folder, which we call the Tutor "environment", and which contains all the files that are necessary to run an Open edX platform: these are mostly Open edX configuration files.
$ cd "$(tutor config printroot)"
$ ls
config.yml env
All these files are stored in a single folder, called the Tutor project root. On Linux, this folder is in ``~/.local/share/tutor``. On Mac OS it is ``~/Library/Application Support/tutor``.
The ``config.yml`` file contains your user-specific Open edX settings (item #1 above). The ``env/`` folder contains the rendered templates which will be used to run your Open edX platform (item #2). For instance, the ``env/local`` folder contains the ``docker-compose.yml`` file to run Open edX locally.
The values from ``config.yml`` are used to generate the environment files in ``env/``. As a consequence, **every time the values from**``config.yml``**are modified, the environment must be regenerated**. This can be done with::
The values from ``config.yml`` are used to generate the environment files in ``env/``. As a consequence, **every time the values from**``config.yml``**are modified, the environment must be regenerated** with ``tutor config save``..
tutor config save
BecausetheTutorenvironmentis generated entirely from the values in ``config.yml``, you can ``rm -rf`` the ``env/`` folder at any time and re-create it with ``tutor config save``. Another consequence is that **any manual change made to a file in**``env/``**will be overwritten by**``tutor config save``**commands**. Consider yourself warned!
Another consequence is that **any manual change made to a file in**``env/``**will be overwritten by**``tutor config save``**commands**. Consider yourself warned!
You can now take advantage of the Tutor-powered CLI (item #3) to bootstrap your Open edX platform::
tutor local launch
Under the hood, Tutor simply runs ``docker compose`` and ``docker`` commands to launch your platform. These commands are printed in the standard output, such that you are free to replicate the same behaviour by simply copying/pasting the same commands.
How do I navigate Tutor's command-line interface?
-------------------------------------------------
Tutor commands are structured in an easy-to-follow hierarchy. At the top level, there are command trees for image and configuration management::
tutor config ...
tutor images ...
as well as command trees for each mode in which Tutor can run::
tutor local ... # Commands for managing a local Open edX deployment.
tutor k8s ... # Commands for managing a Kubernetes Open edX deployment.
tutor dev ... # Commands for hacking on Open edX in development mode.
Within each mode, Tutor has subcommands for managing that type of Open edX instance. Many of them are common between modes, such as ``launch``, ``start``, ``stop``, ``exec``, and ``logs``. For example::
tutor local logs # View logs of a local deployment.
tutor k8s logs # View logs of a Kubernetes-managed deployment.
tutor dev logs # View logs of a development platform.
Many commands can be further parameterized to specify their target and options, for example::
tutor local logs cms # View logs of the CMS container in a local deployment.
tutor k8s logs mysql # View logs of MySQL in Kubernetes-managed deployment.
tutor dev logs lms --tail 10 # View ten lines of logs of the LMS container in development mode.
And that's it! You do not need to understand Tutor's entire command-line interface to get started. Using the ``--help`` option that's available on every command, it is easy to learn as you go. For an in-depth guide, you can also explore the `CLI Reference <reference/index.rst>`_.
@ -20,24 +20,36 @@ Tutor was tested with server version 1.14.1 and client 1.14.3.
Memory
~~~~~~
In the following, we assume you have access to a working Kubernetes cluster. `kubectl` should use your cluster configuration by default. To launch a cluster locally, you may try out Minikube. Just follow the `official installation instructions <https://kubernetes.io/docs/setup/minikube/>`_.
In the following, we assume you have access to a working Kubernetes cluster. ``kubectl`` should use your cluster configuration by default. To launch a cluster locally, you may try out Minikube. Just follow the `official installation instructions <https://kubernetes.io/docs/setup/minikube/>`__.
The Kubernetes cluster should have at least 4Gb of RAM on each node. When running Minikube, the virtual machine should have that muchallocated memory. See below for an example with VirtualBox:
The Kubernetes cluster should have at least 4Gb of RAM on each node. When running Minikube, the virtual machine should have that much-allocated memory. See below for an example with VirtualBox:
..image:: img/virtualbox-minikube-system.png
:alt:Virtualbox memory settings for Minikube
Ingress controller and SSL/TLS certificates
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Load Balancer and SSL/TLS certificates
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
As of Tutor v11, it is no longer required to setup an Ingress controller to access your platform. Instead Caddy exposes a LoadBalancer service and SSL/TLS certificates are transparently generated at runtime.
By default, Tutor deploys a `LoadBalancer <https://kubernetes.io/docs/concepts/services-networking/service/#loadbalancer>`__ service that exposes the Caddy deployment to the outside world. As in the local installation, this service is responsible for transparently generating SSL/TLS certificates at runtime. You will need to point your DNS records to this LoadBalancer object before the platform can work correctly. Thus, you should first start the Caddy load balancer, with::
S3-like object storage with `MinIO <https://www.minio.io/>`_
Like many web applications, Open edX needs to persist data. In particular, it needs to persist files uploaded by students and course designers. In the local installation, these files are persisted to disk, on the host filesystem. But on Kubernetes, it is difficult to share a single filesystem between different pods. This would require persistent volume claims with `ReadWriteMany` access mode, and these are difficult to setup.
Get the external IP of this service::
Luckily, there is another solution: at `edx.org <edx.org>`_, uploaded files are persisted on AWS S3: Open edX is compatible out-of-the-box with the S3 API for storing user-generated files. The problem with S3 is that it introduces a dependency on AWS. To solve this problem, Tutor comes with a plugin that emulates the S3 API but stores files on premises. This is achieved thanks to `MinIO <https://www.minio.io/>`_. If you want to deploy a production platform to Kubernetes, you will most certainly need to enable the ``minio`` plugin::
kubectl --namespace openedx get services/caddy
Use this external IP to configure your DNS records. Once the DNS records are configured, you should verify that the Caddy container has properly generated the SSL/TLS certificates by checking the container logs::
tutor k8s logs -f caddy
If for some reason, you would like to deploy your own load balancer, you should set ``ENABLE_WEB_PROXY=false`` just like in the :ref:`local installation <web_proxy>`. Then, point your load balancer at the "caddy" service, which will be a `ClusterIP <https://kubernetes.io/docs/concepts/services-networking/service/#publishing-services-service-types>`__.
S3-like object storage with `MinIO <https://www.minio.io/>`__
Like many web applications, Open edX needs to persist data. In particular, it needs to persist files uploaded by students and course designers. In the local installation, these files are persisted to disk, on the host filesystem. But on Kubernetes, it is difficult to share a single filesystem between different pods. This would require persistent volume claims with `ReadWriteMany` access mode, and these are difficult to set up.
Luckily, there is another solution: at `edx.org <edx.org>`_, uploaded files are persisted on AWS S3: Open edX is compatible out-of-the-box with the S3 API for storing user-generated files. The problem with S3 is that it introduces a dependency on AWS. To solve this problem, Tutor comes with a plugin that emulates the S3 API but stores files on-premises. This is achieved thanks to `MinIO <https://www.minio.io/>`__. If you want to deploy a production platform to Kubernetes, you will most certainly need to enable the ``minio`` plugin::
tutor plugins enable minio
@ -46,18 +58,22 @@ The "minio.LMS_HOST" domain name will have to point to your Kubernetes cluster.
Kubernetes dashboard
~~~~~~~~~~~~~~~~~~~~
This is not a requirement per se, but it's very convenient to have a visual interface of the Kubernetes cluster. We suggest the official `Kubernetes dashboard <https://github.com/kubernetes/dashboard/>`_. Depending on your Kubernetes provider, you may need to install a dashboard yourself. There are generic instructions on the `project's README <https://github.com/kubernetes/dashboard/blob/master/README.md>`_. AWS provides `specific instructions <https://docs.aws.amazon.com/eks/latest/userguide/dashboard-tutorial.html>`_.
This is not a requirement per se, but it's very convenient to have a visual interface of the Kubernetes cluster. We suggest the official `Kubernetes dashboard <https://github.com/kubernetes/dashboard/>`__. Depending on your Kubernetes provider, you may need to install a dashboard yourself. There are general instructions on the `project's README <https://github.com/kubernetes/dashboard/blob/master/README.md>`__. AWS provides `specific instructions <https://docs.aws.amazon.com/eks/latest/userguide/dashboard-tutorial.html>`__.
On Minikube, the dashboard is already installed. To access the dashboard, run::
minikube dashboard
Lastly, Tutor itself provides a rudimentary listing of your cluster's nodes, workloads, and services::
tutor k8s status
Technical details
-----------------
Under the hood, Tutor wraps ``kubectl`` commands to interact with the cluster. The various commands called by Tutor are printed in the console, so that you can reproduce and modify them yourself.
Under the hood, Tutor wraps ``kubectl`` commands to interact with the cluster. The various commands called by Tutor are printed in the console so that you can reproduce and modify them yourself.
Basically, the whole platform is described in manifest files stored in ``$(tutor config printroot)/env/k8s``. There is also a ``kustomization.yml`` file at the project root for `declarative application management <https://kubectl.docs.kubernetes.io/pages/app_management/apply.html>`_. This allows us to start and update resources with commands similar to ``kubectl apply -k $(tutor config printroot) --selector=...`` (see the ``kubectl apply```official documentation <https://kubectl.docs.kubernetes.io/pages/app_management/apply.html>`_).
Basically, the whole platform is described in manifest files stored in ``$(tutor config printroot)/env/k8s``. There is also a ``kustomization.yml`` file at the project root for `declarative application management <https://kubectl.docs.kubernetes.io/guides/config_management/introduction/#declarative-application-management>`__. This allows us to start and update resources with commands similar to ``kubectl apply -k $(tutor config printroot) --selector=...`` (see the ``kubectl apply```official documentation <https://kubectl.docs.kubernetes.io/references/kubectl/apply/>`__).
The other benefit of ``kubectl apply`` is that it allows you to customise the Kubernetes resources as much as you want. For instance, the default Tutor configuration can be extended by a ``kustomization.yml`` file stored in ``$(tutor config printroot)/env-custom/`` and which would start with::
@ -67,21 +83,21 @@ The other benefit of ``kubectl apply`` is that it allows you to customise the Ku
- ../env/
...
To learn more about "kustomizations", refer to the `official documentation <https://kubectl.docs.kubernetes.io/pages/app_customization/introduction.html>`__.
To learn more about "kustomizations", refer to the `official documentation <https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/>`__.
Quickstart
----------
Launch the platform on Kubernetes in one command::
tutor k8s quickstart
tutor k8s launch
All Kubernetes resources are associated to the "openedx" namespace. If you don't see anything in the Kubernetes dashboard, you are probably looking at the wrong namespace... 😉
All Kubernetes resources are associated with the "openedx" namespace. If you don't see anything in the Kubernetes dashboard, you are probably looking at the wrong namespace... 😉
..image:: img/k8s-dashboard.png
:alt:Kubernetes dashboard ("openedx" namespace)
The same ``tutor k8s quickstart`` command can be used to upgrade the cluster to the latest version.
The same ``tutor k8s launch`` command can be used to upgrade the cluster to the latest version.
Other commands
--------------
@ -90,15 +106,24 @@ As with the :ref:`local installation <local>`, there are multiple commands to ru
tutor k8s -h
In particular, the `tutor k8s start` command restarts and reconfigures all services by running ``kubectl apply``. That means that you can delete containers, deployments or just any other kind of resources, and Tutor will re-create them automatically. You should just beware of not deleting any persistent data stored in persistent volume claims. For instance, to restart from a "blank slate", run::
In particular, the ``tutor k8s start`` command restarts and reconfigures all services by running ``kubectl apply``. That means that you can delete containers, deployments, or just any other kind of resources, and Tutor will re-create them automatically. You should just beware of not deleting any persistent data stored in persistent volume claims. For instance, to restart from a "blank slate", run::
tutor k8s stop
tutor k8s start
All non-persisting data will be deleted, and then re-created.
Guides
------
Common tasks
------------
Executing commands inside service pods
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The Tutor and plugin documentation usually often instructions to execute some ``tutor local run ...`` commands. These commands are only valid when running Tutor locally with docker compose, and will not work on Kubernetes. Instead, you should run ``tutor k8s exec ...`` commands. Arguments and options should be identical.
For instance, to run a Python shell in the lms container, run::
tutor k8s exec lms ./manage.py lms shell
Running a custom "openedx" Docker image
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@ -112,8 +137,33 @@ Some Tutor plugins and customization procedures require that the "openedx" image
Updating docker images
~~~~~~~~~~~~~~~~~~~~~~
Kubernetes does not provide a single command for updating docker images out of the box. A `commonly used trick <https://github.com/kubernetes/kubernetes/issues/33664>`_ is to modify an innocuous label on all resources::
Kubernetes does not provide a single command for updating docker images out of the box. A `commonly used trick <https://github.com/kubernetes/kubernetes/issues/33664>`__ is to modify an innocuous label on all resources::
Plugins can customize any Kubernetes resource in Tutor by overriding the definition of the resource with a :patch:`k8s-override` patch. For example, to change the volume size for MongoDB from ``5Gi`` to ``10Gi``, add the following to the plugin:
This method is for deploying Open edX locally on a single server, where docker images are orchestrated with `docker-compose <https://docs.docker.com/compose/overview/>`_.
..note::
As of v16.0.0, Tutor now uses the ``docker compose`` subcommand instead of the separate ``docker-compose`` command.
.._tutor_root:
In the following, environment and data files will be generated in a user-specific project folder which will be referred to as the "**project root**". On Linux, the default project root is ``~/.local/share/tutor``. An alternative project root can be defined by passing the ``--root=...`` option to the ``tutor`` command, or defining the ``TUTOR_ROOT=...`` environment variable::
tutor --root=/path/to/tutorroot run ...
@ -12,24 +17,21 @@ In the following, environment and data files will be generated in a user-specifi
export TUTOR_ROOT=/path/to/tutorroot
tutor run ...
..note::
As of v10.0.0, a locally-running Open edX platform can no longer be accessed from http://localhost or http://studio.localhost. Instead, when running ``tutor local quickstart``, you must now decide whether you are running a platform that will be used in production. If not, the platform will be automatically be bound to http://local.overhang.io and http://studio.local.overhang.io, which are domain names that point to 127.0.0.1 (localhost). This change was made to facilitate internal communication between Docker containers.
Main commands
-------------
All available commands can be listed by running::
tutor local help
tutor local --help
All-in-one command
~~~~~~~~~~~~~~~~~~
A fully-functional platform can be configured and run in one command::
tutor local quickstart
tutor local launch
But you may want to run commands one at a time: it's faster when you need to run only part of the local deployment process, and it helps you understand how your platform works. In the following we decompose the ``quickstart`` command.
But you may want to run commands one at a time: it's faster when you need to run only part of the local deployment process, and it helps you understand how your platform works. In the following, we decompose the ``launch`` command.
Configuration
~~~~~~~~~~~~~
@ -38,7 +40,7 @@ Configuration
tutor config save --interactive
This is the only non-automatic step in the installation process. You will be asked various questions about your Open edX platform and appropriate configuration files will be generated. If you would like to automate this step then you should run ``tutor config save --interactive`` once. After that, there will be a ``config.yml`` file at the root of the project folder: this file contains all the configuration values for your platform, such as randomly generated passwords, domain names, etc.
This is the only non-automatic step in the installation process. You will be asked various questions about your Open edX platform and appropriate configuration files will be generated. If you would like to automate this step then you should run ``tutor config save --interactive`` once. This will generate a ``config.yml`` file in the **project root**. This file contains all the configuration values for your platform, such as randomly generated passwords, domain names, etc. The location of the **project root** can be found by running ``tutor config printroot``. See :ref:`section above <tutor_root>`.
If you want to run a fully automated installation, upload the ``config.yml`` file to wherever you want to run Open edX. You can then entirely skip the configuration step.
@ -75,7 +77,7 @@ Service initialisation
::
tutor local init
tutor local do init
This command should be run just once. It will initialise all applications in a running platform. In particular, this will create the required databases tables and apply database migrations for all applications.
@ -96,9 +98,17 @@ Finally, tracking logs that store `user events <https://edx.readthedocs.io/proje
Notice the **State** column in the output. It will tell you whether each container is starting, restarting, running (``Up``), cleanly stopped (``Exit 0``), or stopped on error (``Exit N``, where N ≠ 0).
Common tasks
------------
.._createuser:
@ -107,18 +117,18 @@ Creating a new user with staff and admin rights
You will most certainly need to create a user to administer the platform. Just run::
tutor local createuser --staff --superuser yourusername user@email.com
tutor local do createuser --staff --superuser yourusername user@email.com
You will asked to set the user password interactively.
You will be asked to set the user password interactively.
.._democourse:
Importing the demo course
~~~~~~~~~~~~~~~~~~~~~~~~~
After a fresh installation, your platform will not have a single course. To import the `Open edX demo course <https://github.com/edx/edx-demo-course>`_, run::
After a fresh installation, your platform will not have a single course. To import the `Open edX demo course <https://github.com/openedx/edx-demo-course>`_, run::
tutor local importdemocourse
tutor local do importdemocourse
.._settheme:
@ -127,9 +137,7 @@ Setting a new theme
The default Open edX theme is rather bland, so Tutor makes it easy to switch to a different theme::
Notice that we pass the hostnames of the LMS and the CMS to the ``settheme`` command: this is because in Open edX, themes are assigned per domain name.
tutor local do settheme mytheme
Out of the box, only the default "open-edx" theme is available. We also developed `Indigo, a beautiful, customizable theme <https://github.com/overhangio/indigo>`__ which is easy to install with Tutor.
@ -152,138 +160,14 @@ After modifying Open edX settings, for instance when running ``tutor config save
tutor local exec lms reload-uwsgi
.._portainer:
Docker container web UI with `Portainer <https://portainer.io/>`__
Portainer is a web UI for managing docker containers. It lets you view your entire Open edX platform at a glace. Try it! It's really cool::
You might want to customise the docker-compose services listed in ``$(tutor config printroot)/env/local/docker-compose.yml``. To do so, you should create a ``docker-compose.override.yml`` file in that same folder::
The values in this file will override the values from ``docker-compose.yml`` and ``docker-compose.prod.yml``, as explained in the `docker-compose documentation <https://docs.docker.com/compose/extends/#adding-and-overriding-configuration>`__.
You can then view the portainer UI at `http://localhost:9000 <http://localhost:9000>`_. You will be asked to define a password for the admin user. Then, select a "Local environment" to work on; hit "Connect" and select the "local" group to view all running containers.
Among many other things, you'll be able to view the logs for each container, which is really useful.
Guides
------
.._web_proxy:
Running Open edX behind a web proxy
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The containerized web server ([Caddy](caddyserver.com/)) needs to listen to ports 80 and 443 on the host. If there is already a webserver running on the host, such as Apache or Nginx, the caddy container will not be able to start. Tutor supports running behind a web proxy. To do so, add the following configuration::
tutor config save --set RUN_CADDY=false --set NGINX_HTTP_PORT=81
In this example, the nginx container port would be mapped to 81 instead of 80. You must then configure the web proxy on the host. As of v11.0.0, configuration files are no longer provided for automatic configuration of your web proxy. Basically, you should setup a reverse proxy to `localhost:NGINX_HTTP_PORT` from the following hosts: LMS_HOST, preview.LMS_HOST, CMS_HOST, as well as any additional host exposed by your plugins.
..warning::
In this setup, the Nginx HTTP port will be exposed to the world. Make sure to configure your server firewall to block unwanted connections to your server's `NGINX_HTTP_PORT`. Alternatively, you can configure the Nginx container to accept only local connections::
tutor config save --set NGINX_HTTP_PORT=127.0.0.1:81
Running multiple Open edX platforms on a single server
With Tutor, it is easy to run multiple Open edX instances on a single server. To do so, the following configuration parameters must be different for all platforms:
- ``TUTOR_ROOT``: so that configuration, environment and data are not mixed up between platforms.
- ``LOCAL_PROJECT_NAME``: the various docker-compose projects cannot share the same name.
- ``NGINX_HTTP_PORT``: ports cannot be shared by two different containers.
- ``LMS_HOST``, ``CMS_HOST``: the different platforms must be accessible from different domain (or subdomain) names.
In addition, a web proxy must be setup on the host, as described :ref:`above <web_proxy>`.
As an example, here is how to launch two different platforms, with nginx running as a web proxy::
# platform 1
export TUTOR_ROOT=~/openedx/site1
tutor config save --interactive --set RUN_CADDY=false --set LOCAL_PROJECT_NAME=tutor_site1 --set NGINX_HTTP_PORT=81
The default settings module loaded by ``edx-platform`` is ``tutor.production``. The folders ``$(tutor config printroot)/env/apps/openedx/settings/lms`` and ``$(tutor config printroot)/env/apps/openedx/settings/cms`` are mounted as ``edx-platform/lms/envs/tutor`` and ``edx-platform/cms/envs/tutor`` inside the docker containers. Thus, to use your own settings, you must do two things:
1. Copy your settings files for the lms and the cms to ``$(tutor config printroot)/env/apps/openedx/settings/lms/mysettings.py`` and ``$(tutor config printroot)/env/apps/openedx/settings/cms/mysettings.py``.
2. Load your settings by adding ``TUTOR_EDX_PLATFORM_SETTINGS=tutor.mysettings`` to ``$(tutor config printroot)/env/local/.env``.
Of course, your settings should be compatible with the docker installation. You can get some inspiration from the ``production.py`` settings modules generated by Tutor, or just import it as a base by adding ``from .production import *`` at the top of ``mysettings.py``.
Upgrading from earlier versions
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Upgrading from v3+
******************
Just upgrade Tutor using your :ref:`favorite installation method <install>` and run quickstart again::
tutor local quickstart
Upgrading from v1 or v2
***********************
Versions 1 and 2 of Tutor were organized differently: they relied on many different ``Makefile`` and ``make`` commands instead of a single ``tutor`` executable. To migrate from an earlier version, you should first stop your platform::
make stop
Then, install Tutor using one of the :ref:`installation methods <install>`. Then, create the Tutor project root and move your data::
mkdir -p "$(tutor config printroot)"
mv config.json data/ "$(tutor config printroot)"
Finally, launch your platform with::
tutor local quickstart
Backups/Migrating to a different server
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
With Tutor, all data are stored in a single folder. This means that it's extremely easy to migrate an existing platform to a different server. For instance, it's possible to configure a platform locally on a laptop, and then move this platform to a production server.
1. Make sure `tutor` is installed on both servers with the same version.
2. Stop any running platform on server 1::
tutor local stop
3. Transfer the configuration, environment and platform data from server 1 to server 2::
4. On server 2, move the data to the right location::
mv /tmp/tutor "$(tutor config printroot)"
5. Start the instance with::
tutor local start -d
Making database dumps
~~~~~~~~~~~~~~~~~~~~~
To dump all data from the MySQL and Mongodb databases used on the platform, run the following commands::
tutor local exec -e MYSQL_ROOT_PASSWORD="$(tutor config printvalue MYSQL_ROOT_PASSWORD)" mysql \
sh -c 'mysqldump --all-databases --password=$MYSQL_ROOT_PASSWORD > /var/lib/mysql/dump.sql'
tutor local exec mongodb mongodump --out=/data/db/dump.mongodb
The ``dump.sql`` and ``dump.mongodb`` files will be located in ``$(tutor config printroot)/data/mysql`` and ``$(tutor config printroot)/data/mongodb``.
Similarly, the job service configuration can be overridden by creating a ``docker-compose.jobs.override.yml`` file in that same folder.
Tutor comes with a plugin system that allows anyone to customise the deployment of an Open edX platform very easily. The vision behind this plugin system is that users should not have to fork the Tutor repository to customise their deployments. For instance, if you have created a new application that integrates with Open edX, you should not have to describe how to manually patch the platform settings, ``urls.py`` or ``*.env.json`` files. Instead, you can create a "tutor-myapp" plugin for Tutor. Then, users will start using your application in three simple steps::
# 1) Install the plugin
pip install tutor-myapp
# 2) Enable the plugin
tutor plugins enable myapp
# 3) Reconfigure and restart the platform
tutor local quickstart
For simple changes, it may be extremely easy to create a Tutor plugin: even non-technical users may get started with :ref:`simple yaml plugins <plugins_yaml>`.
In the following we learn how to use and create Tutor plugins.
Commands
--------
List installed plugins::
tutor plugins list
Enable/disable a plugin::
tutor plugins enable myplugin
tutor plugins disable myplugin
After enabling or disabling a plugin, the environment should be re-generated with::
tutor config save
.._existing_plugins:
Existing plugins
----------------
Officially-supported plugins are listed on the `Overhang.IO <https://overhang.io/tutor/plugins>`__ website.
Please be aware that as of May 2023 Google Analytics support has been upgraded from Google Universal Analytics to Google Analytics 4 and you may need to update your configuration as mentioned in the `Open edX docs <https://docs.openedx.org/en/latest/site_ops/how-tos/google-analytics.html>`__.
Tutor comes with a plugin system that allows anyone to customise the deployment of an Open edX platform very easily. The vision behind this plugin system is that users should not have to fork the Tutor repository to customise their deployments. For instance, if you have created a new application that integrates with Open edX, you should not have to describe how to manually patch the platform settings, ``urls.py`` or ``*.env.yml`` files. Instead, you can create a "tutor-myapp" plugin for Tutor. This plugin will be in charge of making changes to the platform settings. Then, users will be able to use your application in three simple steps::
# 1) Install the plugin
pip install tutor-myapp
# 2) Enable the plugin
tutor plugins enable myapp
# 3) Reconfigure and restart the platform
tutor local launch
For simple changes, it may be extremely easy to create a Tutor plugin: even non-technical users may get started with our :ref:`plugin_development_tutorial` tutorial. We also provide a list of :ref:`simple example plugins <plugins_examples>`.
To learn about the different ways in which plugins can extend Tutor, check out the :ref:`hooks catalog <hooks_catalog>`.
Plugin commands cheatsheet
==========================
List installed plugins::
tutor plugins list
Enable/disable a plugin::
tutor plugins enable myplugin
tutor plugins disable myplugin
The full plugins CLI is described in the :ref:`reference documentation <cli_plugins>`.
.._existing_plugins:
Existing plugins
================
Many plugins are available from plugin indexes. These indexes are lists of plugins, similar to the `pypi <https://pypi.org>`__ or `npm <npmjs.com/>`__ indexes. By default, Tutor comes with the "main" plugin index. You can check available plugins from this index by running::
tutor plugins update
tutor plugins search
More plugins can be downloaded from the "contrib" index::
tutor plugins index add contrib
tutor plugins search
The "main" and "contrib" indexes include a curated list of plugins that are well maintained and introduce useful features to Open edX. These indexes are maintained by `Overhang.IO <https://overhang.io>`__. For more information about these indexes, refer to the official `overhangio/tpi <https://github.com/overhangio/tpi>`__ repository.
Thanks to these indexes, it is very easy to download and upgrade plugins. For instance, to install the `notes plugin <https://github.com/overhangio/tutor-notes/>`__::
tutor plugins install notes
Upgrade all your plugins with::
tutor plugins upgrade all
To list indexes that you are downloading plugins from, run::
tutor plugins index list
For more information about these indexes, check the `official Tutor plugin indexes (TPI) <https://github.com/overhangio/tpi/>`__ repository.
Plugins can affect the behaviour of Tutor at multiple levels. First, plugins can define new services with their Docker images, settings and the right initialisation commands. To do so you will have to define custom :ref:`config <plugin_config>`, :ref:`patches <plugin_patches>`, :ref:`hooks <plugin_hooks>` and :ref:`templates <plugin_templates>`. Then, plugins can also extend the CLI by defining their own :ref:`commands <plugin_command>`.
..include:: legacy.rst
.._plugin_config:
Plugins can affect the behaviour of Tutor at multiple levels. They can:
* Add new settings or modify existing ones in the Tutor configuration (see :ref:`config <v0_plugin_config>`).
* Add new templates to the Tutor project environment or modify existing ones (see :ref:`patches <v0_plugin_patches>`, :ref:`templates <v0_plugin_templates>` and :ref:`hooks <v0_plugin_hooks>`).
* Add custom commands to the Tutor CLI (see :ref:`command <v0_plugin_command>`).
There exist two different APIs to create Tutor plugins: either with YAML files or Python packages. YAML files are more simple to create but are limited to just configuration and template patches.
.._v0_plugin_config:
config
~~~~~~
The ``config`` attribute is used to modify existing and add new configuration parameters:
* ``config["add"]`` are key/values that should be added to the user-specific ``config.yml`` configuration. Add there passwords, secret keys and other values that do not have a default value.
* ``config["defaults"]`` are default key/values for this plugin. These values will not be added to the ``config.yml`` user file unless users override them manually with ``tutor config save --set ...``.
* ``config["set"]`` are existing key/values that should be modified. Be very careful what you add there! Plugins may define conflicting values for some parameters.
* ``config["add"]`` are key/values that should be added to the user-specific ``config.yml`` configuration. Add there the passwords, secret keys, and other values that do not have a reasonable default value for all users.
* ``config["defaults"]`` are default key/values for this plugin. These values can be accessed even though they are not added to the ``config.yml`` user file. Users can override them manually with ``tutor config save --set ...``.
* ``config["set"]`` are existing key/values that should be modified. Be very careful what you add there! Different plugins may define conflicting values for some parameters.
"set" and "default" key names will be automatically prefixed with the plugin name, in upper case.
"add" and "defaults" key names will be automatically prefixed with the plugin name, in upper case.
Example::
@ -36,19 +44,12 @@ This configuration from the "myplugin" plugin will set the following values:
- ``MYPLUGIN_DOCKER_IMAGE``: this value will by default not be stored in ``config.yml``, but ``tutor config printvalue MYPLUGIN_DOCKER_IMAGE`` will print ``username/imagename:latest``.
- ``MASTER_PASSWORD`` will be set to ``h4cked``. Needless to say, plugin developers should avoid doing this.
.._plugin_patches:
.._v0_plugin_patches:
patches
~~~~~~~
Plugin patches affect the rendered environment templates. In many places the Tutor templates include calls to ``{{ patch("patchname") }}``. This grants plugin developers the possibility to modify the content of rendered templates. Plugins can add content in these places by adding values to the ``patches`` attribute.
..note::
The list of existing patches can be found by searching for `{{ patch(` strings in the Tutor source code::
git grep "{{ patch"
The list of patches can also be browsed online `on Github <https://github.com/search?utf8=✓&q={{+patch+repo%3Aoverhangio%2Ftutor+path%3A%2Ftutor%2Ftemplates&type=Code&ref=advsearch&l=&l= 8>`__.
Plugin patches affect the rendered environment templates. In many places the Tutor templates include calls to ``{{ patch("patchname") }}``. This grants plugin developers the possibility to modify the content of rendered templates. Plugins can add content in these places by adding values to the ``patches`` attribute. See :ref:`patches` for the complete list available patches.
Example::
@ -60,10 +61,11 @@ Example::
This will add a Redis instance to the services run with ``tutor local`` commands.
..note::
The ``patches`` attribute can be a callable function instead of a static dict value.
In Python plugins, remember that ``patches`` can be a callable function instead of a static dict value.
One can use this to dynamically load a list of patch files from a folder.
.._plugin_hooks:
.._v0_plugin_hooks:
hooks
~~~~~
@ -83,9 +85,9 @@ Example::
During initialisation, "myservice1" and "myservice2" will be run in sequence with the commands defined in the templates ``myplugin/hooks/myservice1/init`` and ``myplugin/hooks/myservice2/init``.
To initialise a "foo" service, Tutor runs the "foo-job" service that is found in the ``env/local/docker-compose.jobs.yml`` file. By default, Tutor comes with a few services in this file: mysql-job, lms-job, cms-job, forum-job. If your plugin requires running custom services during initialisation, you will need to add them to the ``docker-compose.jobs.yml`` template. To do so, just use the "local-docker-compose-jobs-services" patch.
To initialise a "foo" service, Tutor runs the "foo-job" service that is found in the ``env/local/docker-compose.jobs.yml`` file. By default, Tutor comes with a few services in this file: mysql-job, lms-job, cms-job. If your plugin requires running custom services during initialisation, you will need to add them to the ``docker-compose.jobs.yml`` template. To do so, just use the "local-docker-compose-jobs-services" patch.
In Kubernetes, the approach is the same, except that jobs are implemented as actual job objects in the ``k8s/jobs.yml`` template. To add your own services there, your plugin should implement the "k8s-jobs" patch.
In Kubernetes, the approach is the same, except that jobs are implemented as actual job objects in the ``k8s/jobs.yml`` template. To add your services there, your plugin should implement the "k8s-jobs" patch.
``pre-init``
++++++++++++
@ -134,12 +136,12 @@ or::
tutor images pull all
tutor images push all
.._plugin_templates:
.._v0_plugin_templates:
templates
~~~~~~~~~
In order to define plugin-specific hooks, a plugin should also have a template directory that includes the plugin hooks. The ``templates`` attribute should point to that directory.
To define plugin-specific hooks, a plugin should also have a template directory that includes the plugin hooks. The ``templates`` attribute should point to that directory.
Example::
@ -148,14 +150,14 @@ Example::
With the above declaration, you can store plugin-specific templates in the ``templates/myplugin`` folder next to the ``plugin.py`` file.
In Tutor, templates are `Jinja2 <https://jinja.palletsprojects.com/en/2.11.x/>`__-formatted files that will be rendered in the Tutor environment (the ``$(tutor config printroot)/env`` folder) when running ``tutor config save``. The environment files are overwritten every time the environment is saved. Plugin developers can create templates that make use of the built-in `Jinja2 API <https://jinja.palletsprojects.com/en/2.11.x/api/>`__. In addition, a couple additional filters are added by Tutor:
In Tutor, templates are `Jinja2 <https://jinja.palletsprojects.com/en/2.11.x/>`__-formatted files that will be rendered in the Tutor environment (the ``$(tutor config printroot)/env`` folder) when running ``tutor config save``. The environment files are overwritten every time the environment is saved. Plugin developers can create templates that make use of the built-in `Jinja2 API <https://jinja.palletsprojects.com/en/2.11.x/api/>`__. In addition, a couple of additional filters are added by Tutor:
* ``common_domain``: Return the longest common name between two domain names. Example: ``{{ "studio.demo.myopenedx.com"|common_domain("lms.demo.myopenedx.com") }}`` is equal to "demo.myopenedx.com".
* ``encrypt``: Encrypt an arbitrary string. The encryption process is compatible with `htpasswd <https://httpd.apache.org/docs/2.4/programs/htpasswd.html>`__ verification.
* ``list_if``: In a list of ``(value, condition)`` tuples, return the list of ``value`` for which the ``condition`` is true.
* ``long_to_base64``: Base-64 encode a long integer.
* ``iter_values_named``: Yield the values of the configuration settings that match a certain pattern. Example: ``{% for value in iter_values_named(prefix="KEY", suffix="SUFFIX")%}...{% endfor %}``. By default, only non-empty values are yielded. To iterate also on empty values, pass the ``allow_empty=True`` argument.
* ``patch``: See :ref:`patches <plugin_patches>`.
* ``patch``: See :ref:`patches <v0_plugin_patches>`.
* ``random_string``: Return a random string of the given length composed of ASCII letters and digits. Example: ``{{ 8|random_string }}``.
* ``reverse_host``: Reverse a domain name (see `reference <https://en.wikipedia.org/wiki/Reverse_domain_name_notation>`__). Example: ``{{ "demo.myopenedx.com"|reverse_host }}`` is equal to "com.myopenedx.demo".
* ``rsa_import_key``: Import a PEM-formatted RSA key and return the corresponding object.
@ -171,27 +173,37 @@ When saving the environment, template files that are stored in a template root w
* Binary files with the following extensions: .ico, .jpg, .png, .ttf
* Files that are stored in a folder named "partials", or one of its subfolders.
.._plugin_command:
.._v0_plugin_command:
command
~~~~~~~
A plugin can provide custom command line commands. Commands are assumed to be `click.Command <https://click.palletsprojects.com/en/7.x/api/#commands>`__ objects.
Python plugins can provide a custom command line interface.
The ``command`` attribute is assumed to be a `click.Command <https://click.palletsprojects.com/en/8.0.x/api/#commands>`__ object,
and you typically implement them using the `click.command <https://click.palletsprojects.com/en/8.0.x/api/#click.command>`__ decorator.
You may also use the `click.pass_obj <https://click.palletsprojects.com/en/8.0.x/api/#click.pass_obj>`__ decorator to pass the CLI `context <https://click.palletsprojects.com/en/8.0.x/api/#click.Context>`__, such as when you want to access Tutor configuration settings from your command.
Example::
import click
from tutor import config as tutor_config
@click.command(help="I'm a plugin command")
def command():
@click.pass_obj
def command(context):
config = tutor_config.load(context.root)
lms_host = config["LMS_HOST"]
click.echo("Hello from myplugin!")
click.echo(f"My LMS host is {lms_host}")
Any user who installs the ``myplugin`` plugin can then run::
$ tutor myplugin
Hello from myplugin!
My LMS host is learn.myserver.com
You can even define subcommands by creating `command groups <https://click.palletsprojects.com/en/7.x/api/#click.Group>`__::
You can even define subcommands by creating `command groups <https://click.palletsprojects.com/en/8.0.x/api/#click.Group>`__::
import click
@ -199,13 +211,23 @@ You can even define subcommands by creating `command groups <https://click.palle
def command():
pass
@click.command(help="I'm a plugin subcommand")
@command.command(help="I'm a plugin subcommand")
def dosomething():
click.echo("This subcommand is awesome")
This would allow any user to run::
This would allow any user to see your sub-commands::
$ tutor myplugin
Usage: tutor myplugin [OPTIONS] COMMAND [ARGS]...
I'm a plugin command group
Commands:
dosomething I'm a plugin subcommand
and then run them::
$ tutor myplugin dosomething
This subcommand is awesome
See the official `click documentation <https://click.palletsprojects.com/en/7.x/>`__ for more information.
See the official `click documentation <https://click.palletsprojects.com/en/8.0.x/>`__ for more information.
Plugins can be created in two different ways: either as plain YAML files or installable Python packages. YAML files are great when you need to make minor changes to the default platform, such as modifying settings. For creating more complex applications, it is recommended to create python packages.
.._plugins_yaml:
@ -14,7 +16,7 @@ YAML files that are stored in the tutor plugins root folder will be automaticall
On Linux, this points to ``~/.local/share/tutor-plugins``. The location of the plugin root folder can be modified by setting the ``TUTOR_PLUGINS_ROOT`` environment variable.
YAML plugins need to define two extra keys: "name" and "version". Custom CLI commands are not supported by YAML plugins.
YAML plugins must define two special top-level keys: ``name`` and ``version``. Then, YAML plugins may use two more top-level keys to customise Tutor's behaviour: ``config`` and ``patches``. Custom CLI commands, templates, and hooks are not supported by YAML plugins.
Let's create a simple plugin that adds your own `Google Analytics <https://analytics.google.com/>`__ tracking code to your Open edX platform. We need to add the ``GOOGLE_ANALYTICS_ACCOUNT`` and ``GOOGLE_ANALYTICS_TRACKING_ID`` settings to both the LMS and the CMS settings. To do so, we will only have to create the ``openedx-common-settings`` patch, which is shared by the development and the production settings both for the LMS and the CMS. First, create the plugin directory::
@ -39,17 +41,13 @@ You can then enable your newly-created plugin::
tutor plugins enable googleanalytics
Update your environment to apply changes from your plugin::
tutor config save
You should be able to view your changes in every LMS and CMS settings file::
Now just restart your platform to start sending tracking events to Google Analytics::
tutor local quickstart
tutor local launch
That's it! And it's very easy to share your plugins. Just upload them to your Github repo and share the url with other users. They will be able to install your plugin by running::
@ -58,17 +56,34 @@ That's it! And it's very easy to share your plugins. Just upload them to your Gi
Python package
~~~~~~~~~~~~~~
Creating a plugin as a Python package allows you to define more complex logic and to store your patches in a more structured way. Python Tutor plugins are regular Python packages that define a specific entrypoint: ``tutor.plugin.v0``.
Creating a plugin as a Python package allows you to define more complex logic and store your patches in a more structured way. Python Tutor plugins are regular Python packages that define an entry point within the ``tutor.plugin.v0`` group:
The ``myplugin.plugin`` python module should then declare the ``config``, ``hooks``, etc. attributes that will define its behaviour.
The ``myplugin/plugin.py`` Python module can then define the attributes ``config``, ``patches``, ``hooks``, and ``templates`` to specify the plugin's behaviour. The attributes may be defined either as dictionaries or as zero-argument callables returning dictionaries; in the latter case, the callable will be evaluated upon plugin load. Finally, the ``command`` attribute can be defined as an instance of ``click.Command`` to define the plugin's command line interface.
Example::
import click
templates = pkg_resources.resource_filename(...)
config = {...}
hooks = {...}
def patches():
...
@click.command(...)
def command():
...
To get started on the right foot, it is strongly recommended to create your first plugin with the `tutor plugin cookiecutter <https://github.com/overhangio/cookiecutter-tutor-plugin>`__::
..warning:: The v0 plugin API is no longer the recommended way of developing new plugins for Tutor, starting from Tutor v13.2.0. See our :ref:`plugin creation tutorial <plugin_development_tutorial>` to learn more about the v1 plugin API. Existing v0 plugins will remain supported for some time but developers are encouraged to start migrating their plugins as soon as possible to make use of the new API. Please read the `upgrade instructions <https://github.com/overhangio/cookiecutter-tutor-plugin>`__ to upgrade v0 plugins generated with the v0 plugin cookiecutter.
You have the option of running Tutor with `Podman <https://podman.io/>`__, instead of the native Docker tools. This has some practical advantages: it does not require a running Docker daemon, and it enables you to run and build Docker images without depending on any system component running ``root``. As such, it is particularly useful for building Tutor images from CI pipelines.
The ``podman`` CLI aims to be fully compatible with the ``docker`` CLI, and ``podman-compose`` is meant to be a fully-compatible alias of ``docker-compose``. This means that you should be able to use together with Tutor, without making any changes to Tutor itself.
..warning::
Since this was written, it was discovered that there are major compatibility issues between ``podman-compose`` and ``docker-compose``. Thus, podman cannot be considered a drop-in replacement of Docker in the context of Tutor -- at least for running Open edX locally.
..warning::
You should not attempt to run Tutor with Podman on a system that already has native ``docker`` installed. If you want to switch to ``podman`` using the aliases described here, you should uninstall (or at least stop) the native Docker daemon first.
Enabling Podman
~~~~~~~~~~~~~~~
Podman is supported on a variety of development platforms, see the `installation instructions <https://podman.io/getting-started/installation>`_ for details.
Once you have installed Podman and its dependencies on the platform of your choice, you'll need to make sure that its ``podman`` binary, usually installed as ``/usr/bin/podman``, is aliased to ``docker``, and is included as such in your system ``$PATH``. On some CentOS and Fedora releases you can install a package named ``podman-docker`` to do this for you, but on other platforms you'll need to take of this yourself.
- If ``$HOME/bin`` is in your ``$PATH``, you can create a symbolic link there::
ln -s $(which podman) $HOME/bin/docker
- If you want to instead make ``docker`` a system-wide alias for ``podman``, you can create your symlink in ``/usr/local/bin``, an action that normally requires ``root`` privileges::
sudo ln -s $(which podman) /usr/local/bin/docker
Enabling podman-compose
~~~~~~~~~~~~~~~~~~~~~~~
``podman-compose`` is available as a package from PyPI, and can thus be installed with ``pip``. See `its README <https://github.com/containers/podman-compose/blob/devel/README.md>`_ for installation instructions. Note that if you have installed Tutor in its own virtualenv, you'll need to run ``pip install podman-compose`` in that same virtualenv.
Once installed, you'll again need to create a symbolic link that aliases ``docker-compose`` to ``podman-compose``.
- If you run Tutor and ``podman-compose`` in a virtualenv, create the symlink in that virtualenv's ``bin`` directory: activate the virtualenv, then run::
Once you have configured your symbolic links as described, you should be able to run ``docker version`` and ``docker-compose --help`` and their output should agree, respectively, with ``podman version`` and ``podman-compose --help``.
After that, you should be able to use ``tutor local``, ``tutor build``, and other commands as if you had installed the native Docker tools.
1. `Download <https://github.com/overhangio/tutor/releases>`_ the latest stable release of Tutor and place the ``tutor`` executable in your path. From the command line:
1. Install the latest stable release of Tutor from pip:
..include::cli_download.rst
..include:: download/pip.rst
2. Run ``tutor local quickstart``
Or `download <https://github.com/overhangio/tutor/releases>`_ the pre-compiled binary and place the ``tutor`` executable in your path:
..include:: download/binary.rst
2. Run ``tutor local launch``
3. You're done!
**That's it?**
Yes :) This is what happens when you run ``tutor local quickstart``:
Yes :) This is what happens when you run ``tutor local launch``:
1. You answer a few questions about the :ref:`configuration` of your Open edX platform.
2. Configuration files are generated from templates.
3. Docker images are downloaded.
4. Docker containers are provisioned.
5. A full, production-ready Open edX platform (`Lilac<https://edx.readthedocs.io/projects/edx-installing-configuring-and-running/en/open-release-lilac.master/platform_releases/koa.html>`__ release) is run with docker-compose.
5. A full, production-ready Open edX platform (`Quince<https://edx.readthedocs.io/projects/edx-installing-configuring-and-running/en/open-release-quince.master/platform_releases/quince.html>`__ release) is run with docker-compose.
The whole procedure should require less than 10 minutes, on a server with a good bandwidth. Note that your host environment will not be affected in any way, since everything runs inside docker containers. Root access is not even necessary.
The whole procedure should require less than 10 minutes, on a server with good bandwidth. Note that your host environment will not be affected in any way, since everything runs inside docker containers. Root access is not even necessary.
There's a lot more to Tutor than that! To learn more about what you can do with Tutor and Open edX, check out the :ref:`whatnext` section. If the quickstart installation method above somehow didn't work for you, check out the :ref:`troubleshooting` guide.
There's a lot more to Tutor than that! To learn more about what you can do with Tutor and Open edX, check out the :ref:`whatnext` section. If the launch installation method above somehow didn't work for you, check out the :ref:`troubleshooting` guide.
Actions are one of the two types of hooks (the other being :ref:`filters`) that can be used to extend Tutor. Each action represents an event that can occur during the application life cycle. Each action has a name, and callback functions can be attached to it. When an action is triggered, these callback functions are called in sequence. Each callback function can trigger side effects, independently from one another.
..autoclass:: tutor.core.hooks.Action
:members:
.. The following are only to ensure that the docs build without warnings
Tutor can be extended by making use of "hooks". Hooks are either "actions" or "filters". Here, we list all instances of actions and filters that are used across Tutor. Plugin developers can leverage these hooks to modify the behaviour of Tutor.
The underlying Python hook classes and API are documented :ref:`here <hooks_api>`.
Contexts are a feature of the hook-based extension system in Tutor, which allows us to keep track of which components of the code created which callbacks. Contexts are very much an internal concept that most plugin developers should not have to worry about.
Filters are one of the two types of hooks (the other being :ref:`actions`) that can be used to extend Tutor. Filters allow one to modify the application behavior by transforming data. Each filter has a name, and callback functions can be attached to it. When a filter is applied, these callback functions are called in sequence; the result of each callback function is passed as the first argument to the next callback function. The result of the final callback function is returned to the application as the filter's output.
..autoclass:: tutor.core.hooks.Filter
:members:
.. The following are only to ensure that the docs build without warnings
This is the Python documentation of the two types of hooks (actions and filters) as well as the contexts system which is used to instrument them. Understanding how Tutor hooks work is useful to create plugins that modify the behaviour of Tutor. However, plugin developers should almost certainly not import these hook types directly. Instead, use the reference :ref:`hooks catalog <hooks_catalog>`.
Plugin indexes are a great way to have your plugins discovered by other users. Plugin indexes make it easy for other Tutor users to install and upgrade plugins from other developers. Examples include the official indexes, which can be found in the `overhangio/tpi <https://github.com/overhangio/tpi/>`__ repository.
Index file paths
================
A plugin index is a yaml-formatted file. It can be stored on the web or on your computer. In both cases, the index file location must end with "<current release name>/plugins.yml". For instance, the following are valid index locations if you run the Open edX "Quince" release:
To add either indexes, run the ``tutor plugins index add`` command without the suffix. For instance::
tutor plugins index add https://overhang.io/tutor/main
tutor plugins index add /path/to/your/local/index/
Your plugin cache should be updated immediately. You can also update the cache at any point by running::
tutor plugins update
To view current indexes, run::
tutor plugins index list
To remove an index, run::
tutor plugins index remove <index url>
Plugin entry syntax
===================
A "plugins.yml" file is a yaml-formatted list of plugin entries. Each plugin entry has two required fields: "name" and "src". For instance, here is a minimal plugin entry::
- name: mfe
src: tutor-mfe
"name" (required)
-----------------
A plugin name is how it will be referenced when we run ``tutor plugins install <name>`` or ``tutor plugins enable <name>``. It should be concise and easily identifiable, just like a Python or apt package name.
Plugins with duplicate names will be overridden, depending on the index in which they are declared: indexes further down ``tutor plugins index list`` (which have been added later) will have higher priority.
.._plugin_index_src:
"src" (required)
----------------
A plugin source can be either:
1. A pip requirement file format specifier (see `reference <https://pip.pypa.io/en/stable/reference/requirements-file-format/>`__).
2. The path to a Python file on your computer.
3. The URL of a Python file on the web.
In the first case, the plugin will be installed as a Python package. In the other two cases, the plugin will be installed as a single-file plugin.
Link to the plugin project, where users can learn more about it and ask for support.
"author" (optional)
-------------------
Original author of the plugin. Feel free to include your company name and email address here. For instance: "Leather Face <niceguy@happyfamily.com>".
"maintainer" (optional)
-----------------------
Current maintainer of the plugin. Same format as "author".
"description" (optional)
------------------------
Multi-line string that should contain extensive information about your plugin. The full description will be displayed with ``tutor plugins show <name>``. It will also be parsed for a match by ``tutor plugins search <pattern>``. Only the first line will be displayed in the output of ``tutor plugins search``. Make sure to keep the first line below 128 characters.
Examples
========
Manage plugins in development
-----------------------------
Plugin developers and maintainers often want to install local versions of their plugins. They usually achieve this with ``pip install -e /path/to/tutor-plugin``. We can improve that workflow by creating an index for local plugins::
# Create the plugin index directory
mkdir -p ~/localindex/quince/
# Edit the index
vim ~/localindex/quince/plugins.yml
Add the following to the index::
- name: myplugin1
src: -e /path/to/tutor-myplugin1
- name: myplugin2
src: -e /path/to/tutor-myplugin2
Then add the index::
tutor plugins index add ~/localindex/
Install the plugins::
tutor plugins install myplugin1 myplugin2
Re-install all plugins::
tutor plugins upgrade all
The latter commands will install from the local index, and not from the remote indexes, because indexes that are added last have higher priority when plugins with the same names are found.
Install plugins from a private index
------------------------------------
Plugin authors might want to share plugins with a limited number of users. This is for instance the case when a plugin is for internal use only.
First, users should have access to the ``plugins.yml`` file. There are different ways to achieve that:
- Make the index public: after all, it's mostly the plugins which are private.
- Grant access to the index from behind a VPN.
- Hide the index behing a basic HTTP auth url. The index can then be added with ``tutor plugins index add http://user:password@mycompany.com/index/``.
- Download the index to disk, and then add it from the local path: ``tutor plugins index add ../path/to/index``.
Second, users should be able to install the plugins that are listed in the index. We recommend that the plugins are uploaded to a pip-compatible self-hosted mirror, such as `devpi <https://devpi.net/docs/devpi/devpi/latest/+doc/index.html>`__. Alternatively, packages can be installed from a private Git repository. For instance::
Both examples work because the :ref:`"src" <plugin_index_src>` field supports just any syntax that could also be included in a requirements file installed with ``pip install -r requirements.txt``.
This is the list of all patches used across Tutor (outside of any plugin). Alternatively, you can search for patches in Tutor templates by grepping the source code::
git clone https://github.com/overhangio/tutor
cd tutor
git grep "{{ patch" -- tutor/templates
Or you can list all available patches with the following command::
tutor config patches list
See also `this GitHub search <https://github.com/search?utf8=✓&q={{+patch+repo%3Aoverhangio%2Ftutor+path%3A%2Ftutor%2Ftemplates&type=Code&ref=advsearch&l=&l= 8>`__.
..patch:: caddyfile
``caddyfile``
=============
File: ``apps/caddy/Caddyfile``
Add here Caddy directives to redirect traffic from the outside to your service containers. You should make use of the "proxy" snippet that simplifies configuration and automatically configures logging. Also, make sure to use the ``$default_site_port`` environment variable to make sure that your service will be accessible both when HTTPS is enabled or disabled. For instance::
{{ MYPLUGIN_HOST }}{$default_site_port} {
import proxy "myservice:8000"
}
See the `Caddy reference documentation <https://caddyserver.com/docs/caddyfile>`__ for more information.
Any Kubernetes resource definition in this patch will override the resource defined by Tutor, provided that their names match. See :ref:`Customizing Kubernetes resources <customizing_kubernetes_sources>` for an example.
..patch:: k8s-services
``k8s-services``
================
File: ``k8s/services.yml``
..patch:: k8s-volumes
``k8s-volumes``
===============
File: ``k8s/volumes.yml``
..patch:: kustomization
``kustomization``
=================
File: ``kustomization.yml``
..patch:: kustomization-commonlabels
``kustomization-commonlabels``
==============================
File: ``kustomization.yml``
..patch:: kustomization-configmapgenerator
``kustomization-configmapgenerator``
====================================
File: ``kustomization.yml``
..patch:: kustomization-patches-strategic-merge
``kustomization-patches-strategic-merge``
=========================================
File: ``kustomization.yml``
This can be used to add more Kustomization patches that make use of the `strategic merge mechanism <https://kubernetes.io/docs/tasks/manage-kubernetes-objects/kustomization/#customizing>`__.
Python-formatted LMS settings in development. Values defined here override the values from :patch:`openedx-lms-common-settings` or :patch:`openedx-lms-production-settings`.
..patch:: openedx-lms-production-settings
``openedx-lms-production-settings``
===================================
File: ``apps/openedx/settings/lms/production.py``
Python-formatted LMS settings in production. Values defined here override the values from :patch:`openedx-lms-common-settings`.
``uwsgi-config``
================
File: ``apps/openedx/settings/uwsgi.ini``
A .INI formatted file used to extend or override the uWSGI configuration.
Check the uWSGI documentation for more details about the `.INI format <https://uwsgi-docs.readthedocs.io/en/latest/Configuration.html#ini-files>`__ and the `list of available options <https://uwsgi-docs.readthedocs.io/en/latest/Options.html>`__.
@ -11,11 +11,12 @@ What should you do if you have a problem?
1. Read the error logs that appear in the console. When running a single server platform as daemon, you can view the logs with the ``tutor local logs`` command. (see :ref:`logging` below)
2. Check if your problem already has a solution right here in the :ref:`troubleshooting` section.
3. Search for your problem in the `open and closed Github issues <https://github.com/overhangio/tutor/issues?utf8=%E2%9C%93&q=is%3Aissue>`_.
4. Search for your problem in the `community forums <https://discuss.overhang.io>`__.
5. If, despite all your efforts, you can't solve the problem by yourself, you should discuss it in the `community forums <https://discuss.overhang.io>`__. Please give as much details about your problem as possible! As a rule of thumb, **people will not dedicate more time to solving your problem than you took to write your question**.
6. If you are *absolutely* positive that you are facing a technical issue with Tutor, and not with Open edX, not with your server, not your custom configuration, then, and only then, should you open an issue on `Github <https://github.com/overhangio/tutor/issues/>`__. You *must* follow the instructions from the issue template!!! If you do not follow this procedure, your Github issues will be mercilessly closed 🤯.
4. Search for your problem in the (now legacy) `Tutor community forums <https://discuss.overhang.io>`__.
5. Search for your problem in the `Open edX community forum <https://discuss.openedx.org/>`__.
6. If despite all your efforts, you can't solve the problem by yourself, you should discuss it in the `Open edX community forum <https://discuss.openedx.org>`__. Please give as many details about your problem as possible! As a rule of thumb, **people will not dedicate more time to solving your problem than you took to write your question**. You should tag your topic with "tutor" or the corresponding Tutor plugin name ("tutor-discovery", etc.) in order to notify the maintainers.
7. If you are *absolutely* positive that you are facing a technical issue with Tutor, and not with Open edX, not with your server, not your custom configuration, then, and only then, should you open an issue on `Github <https://github.com/overhangio/tutor/issues/>`__. You *must* follow the instructions from the issue template!!! If you do not follow this procedure, your Github issues will be mercilessly closed 🤯.
Do you need professional assistance with your tutor-managed Open edX platform? Overhang.IO offers online support as part of its `Long Term Support (LTS) offering <https://overhang.io/tutor/pricing>`__.
Do you need professional assistance with your Open edX platform? `Edly <https://edly.io>`__ provides online support as part of its `Open edX installation service <https://edly.io/services/open-edx-installation/>`__.
.._logging:
@ -23,7 +24,7 @@ Logging
-------
..note::
Logs are of paramount importance for debugging Tutor. When asking for help on the `Tutor forums <https://discuss.overhang.io>`__, **you should always include the unedited logs of your app**. You can get those with::
Logs are of paramount importance for debugging Tutor. When asking for help on the `Open edX forum <https://discuss.openedx.org>`__, **you should always include the unedited logs of your app**. You can get those with::
tutor local logs --tail=100 -f
@ -31,9 +32,9 @@ To view the logs from all containers use the ``tutor local logs`` command, which
tutor local logs --follow
To view the logs from just one container, for instance the webserver::
To view the logs from just one container, for instance, the webserver::
tutor local logs --follow nginx
tutor local logs --follow caddy
The last commands produce the logs since the creation of the containers, which can be a lot. Similar to a ``tail -f``, you can run::
@ -43,10 +44,10 @@ If you'd rather use a graphical user interface for viewing logs, you are encoura
.._webserver:
"Cannot start service nginx: driver failed programming external connectivity"
"Cannot start service caddy: driver failed programming external connectivity"
The containerized Nginx needs to listen to ports 80 and 443 on the host. If there is already a webserver, such as Apache or Nginx, running on the host, the nginx container will not be able to start. To solve this issue, check the section on :ref:`how to setup a web proxy <web_proxy>`.
The containerized Caddy needs to listen to ports 80 and 443 on the host. If there is already a webserver, such as Apache, Caddy, or Nginx, running on the host, the caddy container will not be able to start. To solve this issue, check the section on :ref:`how to setup a web proxy <web_proxy>`.
"Couldn't connect to docker daemon"
-----------------------------------
@ -55,24 +56,55 @@ This is not an error with Tutor, but with your Docker installation. This is freq
docker run --rm hello-world
If the above command does not work, you should fix your Docker installation. Some people will suggest to run Docker as root, or with ``sudo``; **do not do this**. Instead, what you should probably do is to add your user to the "docker" group. For more information, check out the `official Docker installation instructions <https://docs.docker.com/install/linux/linux-postinstall/#manage-docker-as-a-non-root-user>`__.
If the above command does not work, you should fix your Docker installation. Some people will suggest running Docker as root, or with ``sudo``; **do not do this**. Instead, what you should probably do is add your user to the "docker" group. For more information, check out the `official Docker installation instructions <https://docs.docker.com/install/linux/linux-postinstall/#manage-docker-as-a-non-root-user>`__.
.._migrations_killed:
"Running migrations... Killed!" / "Command failed with status 137: docker-compose"
Open edX requires at least 4 GB RAM, in particular to run the SQL migrations. If the ``tutor local quickstart`` command dies after displaying "Running migrations", you most probably need to buy more memory or add swap to your machine. On Docker for Mac OS, by default, containers are allocated at most 2 GB of RAM. You should follow `these instructions from the official Docker documentation <https://docs.docker.com/docker-for-mac/#advanced>`__ to allocate at least 4-5 Gb to the Docker daemon.
Open edX requires at least 4 GB RAM, in particular, to run the SQL migrations. If the ``tutor local launch`` command dies after displaying "Running migrations", you most probably need to buy more memory or add swap to your machine.
If migrations were killed halfway, there is a good chance that the MySQL database is in a state that is hard to recover from. The easiest way to recover is simply to delete all the MySQL data and restart the quickstart process. After you have allocated more memory to the Docker daemon, run::
On macOS, by default, Docker allocates at most 2 GB of RAM to containers. ``launch`` tries to check your current allocation and outputs a warning if it can't find a value of at least 4 GB. You should follow `these instructions from the official Docker documentation <https://docs.docker.com/docker-for-mac/#advanced>`__ to allocate at least 4-5 GB to the Docker daemon.
If migrations were killed halfway, there is a good chance that the MySQL database is in a state that is hard to recover from. The easiest way to recover is simply to delete all the MySQL data and restart the launch process. After you have allocated more memory to the Docker daemon, run::
The most common reason this happens is that you are running two different instances of Tutor simultaneously, causing a port conflict between MySQL containers. Tutor will try to prevent you from doing that (for example, it will stop ``local`` containers if you start ``dev`` ones, and vice versa), but it cannot prevent all edge cases. So, as a first step, stop all possible Tutor platform variants::
tutor dev stop
tutor local stop
tutor k8s stop
And then run your command(s) again, ensuring you're consistently using the correct Tutor variant (``tutor dev``, ``tutor local``, or ``tutor k8s``).
If that doesn't work, then check if you have any other Docker containers running that may using port 3306::
docker ps -a
For example, if you have ever used `Tutor Nightly <https://docs.tutor.edly.io/tutorials/nightly.html>`_, check whether you still have ``tutor_nightly_`` containers running. Conversely, if you're trying to run Tutor Nightly now, check whether you have non-Nightly ``tutor_`` containers running. If so, switch to that other version of Tutor, run ``tutor (dev|local|k8s) stop``, and then switch back to your preferred version of Tutor.
Alternatively, if there are any other non-Tutor containers using port 3306, then stop and remove them::
docker stop <container_name>
docker rm <container_name>
Finally, if you've ensured that containers or other programs are making use of port 3306, check the logs of the MySQL container itself::
tutor (dev|local|k8s) logs mysql
Check whether the MySQL container is crashing upon startup, and if so, what is causing it to crash.
Help! The Docker containers are eating all my RAM/CPU/CHEESE
@ -80,6 +112,34 @@ You can identify which containers are consuming most resources by running::
docker stats
In idle mode, the "mysql" container should use ~200MB memory; ~200-300MB for the the "lms" and "cms" containers.
On some operating systems, such as RedHat, Arch Linux or Fedora, a very high limit of the number of open files (``nofile``) per container may cause the "mysql", "lms" and "cms" containers to use a lot of memory: up to 8-16GB. To check whether you might impacted, run::
cat /proc/$(pgrep dockerd)/limits | grep "Max open files"
If the output is 1073741816 or higher, then it is likely that you are affected by `this mysql issue <https://github.com/docker-library/mysql/issues/579>`__. To learn more about the root cause, read `this containerd issue comment <https://github.com/containerd/containerd/pull/7566#issuecomment-1285417325>`__. Basically, the OS is hard-coding a very high limit for the allowed number of open files, and this is causing some containers to fail. To resolve the problem, you should configure the Docker daemon to enforce a lower value, as described `here <https://github.com/docker-library/mysql/issues/579#issuecomment-1432576518>`__. Edit ``/etc/docker/daemon.json`` and add the following contents::
{
"default-ulimits": {
"nofile": {
"Name": "nofile",
"Hard": 1048576,
"Soft": 1048576
}
}
}
Check your configuration is valid with::
dockerd --validate
Then restart the Docker service::
sudo systemctl restart docker.service
Launch your Open edX platform again with ``tutor local launch``. You should observe normal memory usage.
ValueError: Unable to configure handler 'local': [Errno 2] No such file or directory
This will occur if you try to run a development environment without patching the LOGGING configuration, as indicated in the `development_` section. Maybe you correctly patched the development settings, but they are not taken into account? For instance, you might have correctly defined the ``TUTOR_EDX_PLATFORM_SETTINGS`` environment variable, but ``paver`` uses the ``devstack`` settings (which does not patch the ``LOGGING`` variable). This is because calling ``paver lms --settings=development`` or ``paver cms --settings=development`` ignores the ``--settings`` argument. Yes, it might be considered an edx-platform bug... Instead, you should run the ``update_assets`` and ``runserver`` commands, as explained above.
The chosen default language does not display properly
By default, Open edX comes with a `limited set <https://github.com/edx/edx-platform/blob/master/conf/locale/config.yaml>` of translation/localization files. To complement these languages, we add locales from the `openedx-i18n project <https://github.com/openedx/openedx-i18n/blob/master/edx-platform/locale/config-extra.yaml>`_. But not all supported locales are downloaded. In some cases, the chosen default language will not display properly because if was not packaged in either edx-platform or openedx-i18n. If you feel like your language should be packaged, please `open an issue on the openedx-i18n project <https://github.com/openedx/openedx-i18n/issues>`_.
By default, Open edX comes with a `limited set <https://github.com/openedx/edx-platform/blob/master/conf/locale/config.yaml>` of translation/localization files. To complement these languages, we add locales from the `openedx-i18n project <https://github.com/openedx/openedx-i18n/blob/master/edx-platform/locale/config-extra.yaml>`_. But not all supported locales are downloaded. In some cases, the chosen default language will not display properly because it was not packaged in either edx-platform or openedx-i18n. If you feel like your language should be packaged, please `open an issue on the openedx-i18n project <https://github.com/openedx/openedx-i18n/issues>`_.
When I make changes to a course in the CMS, they are not taken into account by the LMS
Yes, there are very few unit tests for now, but this is probably going to change.
Code formatting
---------------
~~~~~~~~~~~~~~~
Tutor code formatting is enforced by `black <https://black.readthedocs.io/en/stable/>`_. To check whether your code changes conform to formatting standards, run::
@ -39,15 +42,18 @@ Static error detection is performed by `pylint <https://pylint.readthedocs.io/en
make test-lint
Bundle ``tutor`` executable
---------------------------
Common developer tasks
----------------------
Generating the ``tutor`` executable binary
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
::
make bundle
Generating the documentation
----------------------------
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
::
@ -60,33 +66,69 @@ You can then browse the documentation with::
make browse
Releasing a new version
-----------------------
~~~~~~~~~~~~~~~~~~~~~~~
- Bump the ``__version__`` value in ``tutor/__about__.py``.
- Replace "Unreleased" by the version name and date in CHANGELOG.md.
- Bump the ``__version__`` value in ``tutor/__about__.py``. (see :ref:`versioning` below)
- Collect changelog entries with ``make changelog``.
- Create a commit with the version changelog.
- Run ``make release``: this will push to the default repo/branch for the current branch.
- Run tests with ``make test``.
- Push your changes to the upstream repository.
.._versioning:
Versioning
----------
The versioning format used in Tutor is the following::
RELEASE.MAJOR.MINOR(-BRANCH)
When making a new Tutor release, increment the:
- RELEASE version when a new Open edX release comes out. The new value should match the ordinal value of the first letter of the release name: Aspen 🡒 1, Birch 🡒 2, ... Zebra 🡒 26.
- MAJOR version when making a backward-incompatible change (prefixed by "💥" in the changelog, as explained below).
- MINOR version when making a backward-compatible change.
An optional BRANCH suffix may be appended to the release name to indicate that extra changes were added on top of the latest release. For instance, "x.y.z-nightly" corresponds to release x.y.z on top of which extra changes were added to make it compatible with the Open edX master branches (see the :ref:`tutorial on running Tutor Nightly <nightly>`).
`Officially-supported plugins <https://overhang.io/tutor/plugins>`__ follow the same versioning pattern. As a third-party plugin developer, you are encouraged to use the same pattern to make it immediately clear to your end-users which Open edX versions are supported.
In Tutor and its officially-supported plugins, certain features, API endpoints, and older depenency versions are periodically deprecated. Generally, warnings are added to the Changelogs and/or the command-line interface one major release before support for any behavior is removed. In order to keep track of pending removals in the source code, comments containing the string ``REMOVE-AFTER-VXX`` should be used, where ``<XX>`` is the last major version that must support the behavior. For example::
# This has been replaced with SOME_NEW_HOOK (REMOVE-AFTER-V25).
SOME_OLD_HOOK = Filter()
indicates that this filter definition can be removed as soon as Tutor v26.0.0.
.._contributing:
Contributing to Tutor
---------------------
Third-party contributions to Tutor and its plugins are more than welcome! Just make sure to follow these guidelines:
Contributions to Tutor and its plugins are highly encouraged. Please adhere to the following guidelines:
- Outside of obvious bugs, contributions should be discussed first in the `official Tutor forums <https://discuss.overhang.io>`__.
- Once we agree on a high-level solution, you should open a pull request on the `Tutor repository <https://github.com/overhangio/tutor/pulls>`__ or the corresponding plugin.
- Write a good Git commit title and message: explain why you are making this change, what problem you are solving and which solution you adopted. Link to the relevant conversation topics in the forums and describe your use case. We *love* long, verbose descriptions :)
- Make sure that all tests pass by running ``make test`` (see above).
- If your PR is in the Tutor core repository, add an item to the CHANGELOG file, in the "Unreleased" section. Use the same format as the other items::
- **General Discussion**: Before addressing anything other than clear-cut bugs, start a discussion on the `official Open edX forum <https://discuss.openedx.org>`__. This facilitates reaching a consensus on a high-level solution.
- **Pull Requests**: For changes to Tutor core or plugin-specific modifications, open a pull request on the `Tutor repository <https://github.com/overhangio/tutor/pulls>`__ or the corresponding plugin repository.
- **Running Tests and Code Formatting**:
- Ensure all tests pass by running ``make test``. This is mandatory for both Tutor core and plugin contributions.
- If formatting tests fail, correct your code format using ``make format``.
- **Changelog Entry**: Create a changelog entry for significant changes (excluding reformatting or documentation) by running ``make changelog-entry``. Edit the newly created file following the given formatting instructions. This applies to both Tutor core and plugin changes.
- **Commit Messages**: Write clear Git commit titles and messages. Detail the rationale for your changes, the issue being addressed, and your solution. Include links to relevant forum discussions and describe your use case. Detailed explanations are valuable. For commit titles, follow `conventional commits <https://www.conventionalcommits.org>`__ guidelines.Additionally, if your pull request addresses an existing GitHub issue, include 'Close #XXX' in your commit message, where XXX is the issue number.
- [TYPE] DESCRIPTION
Releasing a new version
-----------------------
Where "TYPE" is either "Bugfix", "Improvement", "Feature" or "Security". You should add an explosion emoji ("💥") before "[TYPE]" if you are making a breaking change.
When releasing a new version:
- **Version Number**: Update the version number in `__about__.py`. For detailed guidelines on version numbering, refer to the (versioning guidelines :ref:`versioning`).
- **Changelog Compilation**: Compile all changelog entries using ``make changelog``.
- **Git Commit for Release**: Use the format ``git commit -a -m "vX.Y.Z"`` to indicate the new version in the git commit title.
Happy hacking! ☘️
.._maintainers:
Joining the team of Tutor Maintainers
-------------------------------------
We have an open team of volunteers who help support the project. You can read all about it `here <https://discuss.overhang.io/t/the-tutor-maintainer-handbook/1375>`__.
We have an open team of volunteers who help support the project. You can read all about it `here <https://discuss.openedx.org/t/tutor-maintainers/7287>`__ -- and we hope that you'll consider joining us 😉
Tutor can be used on ARM64 systems, and official ARM64 docker images are available starting from Tutor v16.
For older versions of Tutor (v14 or v15), there are several options:
* Use emulation (via qemu or Rosetta 2) to run x86_64 images. Just make sure your installation of Docker supports emulation and use Tutor as normal. This may be 20%-100% slower than native images, depending on the emulation method.
* Use the `unofficial community-maintained ARM64 plugin <https://github.com/open-craft/tutor-contrib-arm64>`_ which will set the required settings for you and which includes unofficial docker images.
* Build your own ARM64 images, e.g. using ``tutor images build openedx permissions`` and/or ``tutor images build openedx-dev`` before launching the LMS.
With Tutor, all data are stored in a single folder. This means that it's extremely easy to migrate an existing platform to a different server. For instance, it's possible to configure a platform locally on a laptop, and then move this platform to a production server.
1. Make sure `tutor` is installed on both servers with the same version.
2. Stop any running platform on server 1::
tutor local stop
3. Transfer the configuration, environment, and platform data from server 1 to server 2::
mysql sh -c 'mysqldump --all-databases --user=$USERNAME --password=$PASSWORD > /var/lib/mysql/dump.sql'
tutor local exec mongodb mongodump --out=/data/db/dump.mongodb
The ``dump.sql`` and ``dump.mongodb`` files will be located in ``$(tutor config printroot)/data/mysql`` and ``$(tutor config printroot)/data/mongodb``.
The default settings module loaded by ``edx-platform`` is ``tutor.production`` in production and ``tutor.development`` in development. The folders ``$(tutor config printroot)/env/apps/openedx/settings/lms`` and ``$(tutor config printroot)/env/apps/openedx/settings/cms`` are mounted as ``edx-platform/lms/envs/tutor`` and ``edx-platform/cms/envs/tutor`` inside the docker containers. To modify these settings you must create a plugin that implements one or more of the patch statements in those setting files. See the :ref:`plugin_development_tutorial` tutorial for more information on how to create a plugin.
Tutor supports running in development with ``tutor dev`` commands. Developers frequently need to work on a fork of some repository. The question then becomes: how to make their changes available within the "openedx" Docker container?
For instance, when troubleshooting an issue in `edx-platform <https://github.com/openedx/edx-platform>`__, we would like to make some changes to a local fork of that repository, and then apply these changes immediately in the "lms" and the "cms" containers (but also "lms-worker", "cms-worker", etc.)
Similarly, when developing a custom XBlock, we would like to hot-reload any change we make to the XBlock source code within the containers.
Tutor provides a simple solution to these questions. In both cases, the solution takes the form of a ``tutor mounts add ...`` command.
Check out the right version of the upstream repository. If you are working on the `current "zebulon" release <https://docs.openedx.org/en/latest/community/release_notes/index.html>`__ of Open edX, then you should checkout the corresponding branch::
# "zebulon" is an example. You should put the actual release name here.
# I.e: aspen, birch, cypress, etc.
git checkout open-release/zebulon.master
On the other hand, if you are working on the Tutor :ref:`"nightly" <nightly>` branch then you should checkout the master branch::
git checkout master
Then, mount the edx-platform repository with Tutor::
tutor mounts add /my/workspace/edx-plaform
This command does a few "magical" things 🧙 behind the scenes:
1. Mount the edx-platform repository in the image at build-time. This means that when you run ``tutor images build openedx``, your custom repository will be used instead of the upstream. In particular, any change you've made to the installed requirements, static assets, etc. will be taken into account.
2. Mount the edx-platform repository at run time. Thus, when you run ``tutor dev start``, any change you make to the edx-platform repository will be hot-reloaded.
You can get a glimpse of how these auto-mounts work by running ``tutor mounts list``. It should output something similar to the following::
Quite often, developers don't want to work on edx-platform directly, but on a dependency of edx-platform. For instance: an XBlock. This works the same way as above. Let's take the example of the `"edx-ora2" <https://github.com/openedx/edx-ora2>`__ package, for open response assessments. First, clone the Python package::
cd /my/workspace/edx-ora2
git clone https://github.com/openedx/edx-ora2 .
Then, check out the right version of the package. This is the version that is indicated in the `edx-platform/requirements/edx/base.txt <https://github.com/openedx/edx-platform/blob/open-release/quince.master/requirements/edx/base.txt>`__. Be careful that the version that is currently in use in your version of edx-platform is **not necessarily the head of the master branch**::
git checkout <my-version-tag-or-branch>
Then, mount this repository::
tutor mounts add /my/workspace/edx-ora2
Verify that your repository is properly bind-mounted by running ``tutor mounts list``::
$ tutor mounts list
- name: /my/workspace/edx-ora2
build_mounts:
- image: openedx
context: mnt-edx-ora2
- image: openedx-dev
context: mnt-edx-ora2
compose_mounts:
- service: lms
container_path: /mnt/edx-ora2
- service: cms
container_path: /mnt/edx-ora2
- service: lms-worker
container_path: /mnt/edx-ora2
- service: cms-worker
container_path: /mnt/edx-ora2
- service: lms-job
container_path: /mnt/edx-ora2
- service: cms-job
container_path: /mnt/edx-ora2
You should then re-build the "openedx" Docker image to pick up your changes::
tutor images build openedx-dev
Then, whenever you run ``tutor dev start``, the "lms" and "cms" container should automatically hot-reload your changes.
To push your changes in production, you should do the same with ``tutor local`` and the "openedx" image::
tutor images build openedx
tutor local start -d
Debugging with breakpoints
--------------------------
To debug a local edx-platform repository, first, start development in detached mode (with ``-d``), add a `python breakpoint <https://docs.python.org/3/library/functions.html#breakpoint>`__ with ``breakpoint()`` anywhere in the code. Then, attach to the applicable service's container by running ``start`` (without ``-d``) followed by the service's name::
# Start in detached mode:
tutor dev start -d
# Debugging LMS:
tutor dev start lms
# Or, debugging CMS:
tutor dev start cms
Running edx-platform unit tests
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
It's possible to run the full set of unit tests that ship with `edx-platform <https://github.com/openedx/edx-platform/>`__. To do so, run a shell in the LMS development container::
tutor dev run lms bash
Then, run unit tests with ``pytest`` commands::
# Run tests on common apps
unset DJANGO_SETTINGS_MODULE
unset SERVICE_VARIANT
export EDXAPP_TEST_MONGO_HOST=mongodb
pytest common
pytest openedx
pytest xmodule
# Run tests on LMS
export DJANGO_SETTINGS_MODULE=lms.envs.tutor.test
pytest lms
# Run tests on CMS
export DJANGO_SETTINGS_MODULE=cms.envs.tutor.test
pytest cms
..note::
Getting all edx-platform unit tests to pass on Tutor is currently a work-in-progress. Some unit tests are still failing. If you manage to fix some of these, please report your findings in the `Open edX forum <https://discuss.openedx.org/tag/tutor>`__.
What if my edx-platform package is not automatically bind-mounted?
It is quite possible that your package is not automatically recognized and bind-mounted by Tutor. Out of the box, Tutor defines a set of regular expressions: if your package name matches this regular expression, it will be automatically bind-mounted. But if it does not, you have to tell Tutor about it.
To do so, you will need to create a :ref:`Tutor plugin <plugin_development_tutorial>` that implements the :py:data:`tutor.hooks.Filters.MOUNTED_DIRECTORIES` filter::
No, you don't. Re-building the "openedx" Docker image may take a while, and you don't want to run this command every time you make a change to your local repositories. Because your host directory is bind-mounted in the containers at runtime, your changes will be automatically applied to the container. If you run ``tutor dev`` commands, then your changes will be automatically picked up.
If you run ``tutor local`` commands (for instance: when debugging a production instance) then your changes will *not* be automatically picked up. In such a case you should manually restart the containers::
tutor local restart lms cms lms-worker cms-worker
Re-building the "openedx" image should only be necessary when you want to push your changes to a Docker registry, then pull them on a remote server.
By default, Tutor comes with a simple SMTP server for sending emails. Such a server has an important limitation: it does not implement mailing good practices, such as DKIM or SPF. As a consequence. the emails you send might be flagged as spam by their recipients. Thus, you might want to disable the SMTP server and run your own, for instance using your Google Mail account.
..warning::
Google Mail SMTP servers come with their own set of limitations. For instance, you are limited to sending 500 emails a day. Reference: https://support.google.com/mail/answer/22839
You should authorize third-party to access your Google Mail account. In your Google Mail account, select "Manage Account", "Security", and turn on "Less Secure App Access". Check the Google documentation for more information on "less secure apps": https://support.google.com/accounts/answer/6010255
Then, check that you can reach the Google Mail SMTP service from your own server::
$ telnet smtp.gmail.com 587
If you get ``Connected to smtp.gmail.com.`` then it means that you can successfully reach the Google Mail SMTP servers. If not, you will have to reconfigure your firewall.
To exit the ``telnet`` shell, type ``ctrl+]``, then ``ctrl+d``.
Then, disable the SMTP server that comes with Tutor::
$ tutor config save --set RUN_SMTP=false
Configure credentials to access your SMTP server::
$ tutor config save \
--set SMTP_HOST=smtp.gmail.com \
--set SMTP_PORT=587 \
--set SMTP_USE_SSL=false \
--set SMTP_USE_TLS=true \
--set SMTP_USERNAME=YOURUSERNAME@gmail.com \
--set SMTP_PASSWORD='YOURPASSWORD'
Don't forget to replace your email address and password in the prompt above. If your email password contains special characters, you might have to escape them.
Then, restart your platform::
$ tutor local launch
That's it! You can send a test email with the following command::
$ tutor local run --no-deps lms ./manage.py lms shell -c \
With Tutor, it is easy to run multiple Open edX instances on a single server. To do so, the following configuration parameters must be different for all platforms:
- ``TUTOR_ROOT``: so that configuration, environment, and data are not mixed up between platforms.
- ``LOCAL_PROJECT_NAME``: the various docker-compose projects cannot share the same name.
- ``CADDY_HTTP_PORT``: exposed ports cannot be shared by two different containers.
- ``LMS_HOST``, ``CMS_HOST``: the different platforms must be accessible from different domain (or subdomain) names.
In addition, a web proxy must be set up on the host, as described :ref:`in the corresponding tutorial <web_proxy>`.
Tutor was designed to make it easy for everyone to run the latest release of Open edX. But sometimes, you want to run the latest, bleeding-edge version of Open edX. This is what we call "running master", as opposed to running the release branch. Running the master branch in production is strongly **not** recommended unless you are an Open edX expert and you really know what you are doing. But Open edX developers frequently need to run the master branch locally to implement and test new features. Thus, Tutor makes it easy to run Open edX on the master branch: this is called "Tutor Nightly".
Installing Tutor Nightly
------------------------
Running Tutor Nightly requires more than setting a few configuration variables: because there are so many Open edX settings, version numbers, etc. which may change between the latest release and the current master branch, Tutor Nightly is actually maintained as a separate branch of the Tutor repository. To install Tutor Nightly, you should install Tutor from the "nightly" branch of the source repository. To do so, run::
As usual, it is strongly recommended to run the command above in a `Python virtual environment <https://docs.python.org/3/tutorial/venv.html>`__.
In addition to installing Tutor Nightly itself, this will install automatically the nightly versions of all official Tutor plugins (which are enumerated in `plugins.txt <https://github.com/overhangio/tutor/tree/nightly/requirements/plugins.txt>`_). Alternatively, if you wish to hack on an official plugin or install a custom plugin, you can clone that plugin's repository and install it. For instance::
Once Tutor Nightly is installed, you can run the usual ``tutor`` commands::
tutor dev launch
tutor dev run lms bash
# ... and so on
Upgrading to the latest version of Open edX
-------------------------------------------
To pull the latest upstream changes, you should first upgrade Tutor Nightly::
cd ./tutor
git pull
Then, you will have to generate a more recent version of the nightly Docker images. Images for running Tutor Nightly are published daily to docker.io (see `here <https://hub.docker.com/r/overhangio/openedx/tags?page=1&ordering=last_updated&name=nightly>`__). You can fetch the latest images with::
tutor images pull all
Alternatively, you may want to build the images yourself. As usual, this is done with::
tutor images build all
However, these images include the application master branch at the point in time when the image was built. The Docker layer caching mechanism might cause the ``git clone`` step from the build to be skipped. In such cases, you will have to bypass the caching mechanism with::
tutor images build --no-cache all
Running Tutor Nightly alongside the latest release
When running Tutor Nightly, you usually do not want to override your existing Tutor installation. That's why a Tutor Nightly installation has the following differences from a regular release installation:
- The default Tutor project root is different in Tutor Nightly. By default it is set to ``~/.local/share/tutor-nightly`` on Linux (instead of ``~/.local/share/tutor``). To modify this location check the :ref:`corresponding documentation <tutor_root>`.
- The plugins root is set to ``~/.local/share/tutor-nightly-plugins`` on Linux (instead of ``~/.local/share/tutor-plugins``). This location may be modified by setting the ``TUTOR_PLUGINS_ROOT`` environment variable.
- The default docker-compose project name is set to ``tutor_nightly_local`` (instead of ``tutor_local``). This value may be modified by manually setting the ``LOCAL_PROJECT_NAME``.
Making changes to Tutor Nightly
-------------------------------
In general pull requests should be open on the "master" branch of Tutor: the "master" branch is automatically merged on the "nightly" branch at every commit, such that changes made to Tutor releases find their way to Tutor Nightly as soon as they are merged. However, sometimes you want to make changes to Tutor Nightly exclusively, and not to the Tutor releases. This might be the case for instance when upgrading the running version of a third-party service (for instance: Elasticsearch, MySQL), or when the master branch requires specific changes. In that case, you should follow the instructions from the :ref:`contributing` section of the docs, with the following differences:
- Open your pull request on top of the "nightly" branch instead of "master".
- Add a description of your changes by creating a changelog entry with `make changelog-entry`, as in the master branch.
Just upgrade Tutor using your :ref:`favorite installation method <install>` and run launch again::
tutor local launch
Upgrading from v1 or v2
~~~~~~~~~~~~~~~~~~~~~~~
Versions 1 and 2 of Tutor were organized differently: they relied on many different ``Makefile`` and ``make`` commands instead of a single ``tutor`` executable. To migrate from an earlier version, you should first stop your platform::
make stop
Then, install Tutor using one of the :ref:`installation methods <install>`. Then, create the Tutor project root and move your data::
Tutor plugins are the officially recommended way of customizing the behaviour of Tutor. If Tutor does not do things the way you want, then your first reaction should *not* be to fork Tutor, but instead to figure out whether you can create a plugin that will allow you to achieve what you want.
You may be thinking that creating a plugin might be overkill for your use case. It's almost certainly not! The stable plugin API guarantees that your changes will keep working even after you upgrade from one major release to the next, with little to no extra work. Also, it allows you to distribute your changes to other users.
A plugin can be created either as a simple, single Python module (a ``*.py`` file) or as a full-blown Python package. Single Python modules are easier to write, while Python packages can be distributed more easily with ``pip install ...``. We'll start by writing our plugin as a single Python module.
Plugins work by making extensive use of the Tutor hooks API. The list of available hooks is available from the :ref:`hooks catalog <hooks_catalog>`. Developers who want to understand how hooks work should check the :ref:`hooks API <hooks_api>`.
Writing a plugin as a single Python module
==========================================
Getting started
---------------
In the following, we'll create a new plugin called "myplugin". We start by creating the plugins root folder::
$ mkdir -p "$(tutor plugins printroot)"
Then, create an empty "myplugin.py" file in this folder::
$ touch "$(tutor plugins printroot)/myplugin.py"
We can verify that the plugin is correctly detected by running::
Our plugin is disabled, for now. To enable it, we run::
$ tutor plugins enable myplugin
Plugin myplugin enabled
Configuration saved to /home/yourusername/.local/share/tutor/config.yml
Environment generated in /home/yourusername/.local/share/tutor/env
At this point your environment was updated, but there would not be any change there... because the plugin does not do anything. So let's get started and make some changes.
Modifying existing files with patches
-------------------------------------
We'll start by modifying some of our Open edX settings files. It's a frequent requirement to modify the ``FEATURES`` setting from the LMS or the CMS in edx-platform. In the legacy native installation, this was done by modifying the ``lms.env.yml`` and ``cms.env.yml`` files. Here we'll modify the Python setting files that define the edx-platform configuration. To achieve that we'll make use of two concepts from the Tutor API: :ref:`patches` and :ref:`filters`.
If you have not already read :ref:`how_does_tutor_work` now would be a good time ☺️ Tutor uses templates to generate various files, such as settings, Dockerfiles, etc. These templates include ``{{ patch("patch-name") }}`` statements that allow plugins to insert arbitrary content in there. These patches are located at strategic locations. See :ref:`patches` for more information.
Let's say that we would like to limit access to our brand new Open edX platform. It is not ready for prime-time yet, so we want to prevent users from registering new accounts. There is a feature flag for that in the LMS: `FEATURES['ALLOW_PUBLIC_ACCOUNT_CREATION'] <https://edx.readthedocs.io/projects/edx-platform-technical/en/latest/featuretoggles.html#featuretoggle-FEATURES['ALLOW_PUBLIC_ACCOUNT_CREATION']>`__. By default this flag is set to a true value, enabling anyone to create an account. In the following we'll set it to false.
Add the following content to the ``myplugin.py`` file that you created earlier::
This imports the ``hooks`` module from Tutor, which grants us access to ``hooks.Actions`` and ``hooks.Filters`` (among other things).
::
hooks.Filters.ENV_PATCHES.add_item(
(
<name>,
<content>
)
)
This means "add ``<content>`` to the ``{{ patch("<name>") }}`` statement, thanks to the :py:data:`tutor.hooks.Filters.ENV_PATCHES` filter". In our case, we want to modify the LMS settings, both in production and development. The right patch for that is :patch:`openedx-lms-common-settings`. We add one item, which is a single Python-formatted line of code::
Your new settings will be taken into account by restarting your platform::
$ tutor local restart
Congratulations! You've created your first working plugin. As you can guess, you can add changes to other files by adding other similar patch statements to your plugin.
Modifying configuration
-----------------------
In the previous section you've learned how to add custom content to the Tutor templates. Now we'll see how to modify the Tutor configuration. Configuration settings can be specified in three ways:
1. "unique" settings that need to be generated or user-specified, and then preserved in config.yml: such settings do not have reasonable defaults for all users. Examples of such settings include passwords and secret keys, which should be different for every user.
2. "default" settings have static fallback values. They are only stored in config.yml when they are modified by users. Most settings belong in this category.
3. "override" settings modify configuration from Tutor core or from other plugins. These will be removed and restored to their default values when the plugin is disabled.
It is very strongly recommended to prefix unique and default settings with the plugin name, in all-caps, such that different plugins with the same configuration do not conflict with one another.
As an example, we'll make it possible to configure public account creation on the LMS via a Tutor setting. In the previous section we achieved that by creating a patch. Let's modify this patch::
hooks.Filters.ENV_PATCHES.add_item(
(
"openedx-lms-common-settings",
"FEATURES['ALLOW_PUBLIC_ACCOUNT_CREATION'] = {% if MYPLUGIN_PLATFORM_IS_PUBLIC %}True{% else %}False{% endif %}",
)
)
This new patch makes use of the ``MYPLUGIN_PLATFORM_IS_PUBLIC`` configuration setting, which we need to create. Since this setting is specific to our plugin and should be stored in config.yml only when it's modified, we create it as a "default" setting. We do that with the :py:data:`tutor.hooks.Filters.CONFIG_DEFAULTS` filter::
hooks.Filters.CONFIG_DEFAULTS.add_item(
("MYPLUGIN_PLATFORM_IS_PUBLIC", False)
)
You can check that the new configuration setting was properly defined::
Now you can quickly toggle the public account creation feature by modifying the new setting::
$ tutor config save --set MYPLUGIN_PLATFORM_IS_PUBLIC=True
$ tutor local restart
Adding new templates
--------------------
If you are adding an extra application to your Open edX platform, there is a good chance that you will create a new Docker image with a custom Dockerfile. This new application will have its own settings and build assets, for instance. This means that you need to add new templates to the Tutor environment. To do that, we will create a new subfolder in our plugins folder::
Create the following Dockerfile in ``$(tutor plugins printroot)/templates/myplugin/build/myservice/Dockerfile``::
FROM docker.io/debian:bullseye-slim
CMD echo "what an awesome plugin!"
Tell Tutor that the "build" folder should be recursively rendered to ``env/plugins/myplugin/build`` with the :py:data:`tutor.hooks.Filters.ENV_TEMPLATE_TARGETS`::
hooks.Filters.ENV_TEMPLATE_TARGETS.add_item(
("myplugin/build", "plugins")
)
At this point you can verify that the Dockerfile template was properly rendered::
We would like to build this image by running ``tutor images build myservice``. For that, we use the :py:data:`tutor.hooks.Filters.IMAGES_BUILD` filter::
hooks.Filters.IMAGES_BUILD.add_item(
(
"myservice", # same name that will be passed to the `build` command
("plugins", "myplugin", "build", "myservice"), # path to the Dockerfile folder
"myservice:latest", # Docker image tag
(), # custom build arguments that will be passed to the `docker build` command
Similarly, to push/pull your image to/from a Docker registry, implement the :py:data:`tutor.hooks.Filters.IMAGES_PUSH` and :py:data:`tutor.hooks.Filters.IMAGES_PULL` filters::
The "myservice" container can be automatically run in local installations by implementing the :patch:`local-docker-compose-services` patch::
hooks.Filters.ENV_PATCHES.add_item(
(
"local-docker-compose-services",
"""
myservice:
image: myservice:latest
"""
)
)
You can now run the "myservice" container which will execute the ``CMD`` statement we wrote in the Dockerfile::
$ tutor config save && tutor local run myservice
...
Creating tutor_local_myservice_run ... done
what an awesome plugin!
Declaring initialisation tasks
------------------------------
Services often need to run specific tasks before they can be started. For instance, the LMS and the CMS need to apply database migrations. These commands are written in shell scripts that are executed whenever we run ``launch``. We call these scripts "init tasks". To add a new local initialisation task, we must first add the corresponding service to the ``docker-compose-jobs.yml`` file by implementing the :patch:`local-docker-compose-jobs-services` patch::
hooks.Filters.ENV_PATCHES.add_item(
(
"local-docker-compose-jobs-services",
"""
myservice-job:
image: myservice:latest
""",
)
)
The patch above defined the "myservice-job" container which will run our initialisation task. Make sure that it is applied by updating your environment::
$ tutor config save
Next, we create an initialisation task by adding an item to the :py:data:`tutor.hooks.Filters.CLI_DO_INIT_TASKS` filter::
hooks.Filters.CLI_DO_INIT_TASKS.add_item(
(
"myservice",
"""
echo "++++++ initialising my plugin..."
echo "++++++ done!"
"""
)
)
Run this initialisation task with::
$ tutor local do init --limit=myplugin
...
Running init task: myplugin/tasks/init.sh
...
Creating tutor_local_myservice-job_run ... done
++++++ initialising my plugin...
++++++ done!
All services initialised.
Tailoring services for development
----------------------------------
When you add services via :patch:`local-docker-compose-services`, those services will be available both in local production mode (``tutor local start``) and local development mode (``tutor dev start``). Sometimes, you may wish to further customize a service in ways that would not be suitable for production, but could be helpful for developers. To add in such customizations, implement the :patch:`local-docker-compose-dev-services` patch. For example, we can enable breakpoint debugging on the "myservice" development container by enabling the ``stdin_open`` and ``tty`` options::
hooks.Filters.ENV_PATCHES.add_item(
(
"local-docker-compose-dev-services",
"""
myservice:
stdin_open: true
tty: true
""",
)
)
Final result
------------
Eventually, our plugin is composed of the following files, all stored within the folder indicated by ``tutor plugins printroot`` (on Linux: ``~/.local/share/tutor-plugins``).
``myplugin.py``
~~~~~~~~~~~~~~~
::
import os
from tutor import hooks
# Define extra folder to look for templates and render the content of the "build" folder
Storing plugins as simple Python modules has the merit of simplicity, but it makes it more difficult to distribute them, either to other users or to remote servers. When your plugin grows more complex, it is recommended to migrate it to a Python package. You should create a package using the `plugin cookiecutter <https://github.com/overhangio/cookiecutter-tutor-plugin>`__. Packages are automatically detected as plugins thanks to the "tutor.plugin.v1" `entry point <https://setuptools.pypa.io/en/latest/userguide/entry_point.html#advertising-behavior>`__. The modules indicated by this entry point will be automatically imported when the plugins are enabled. See the cookiecutter project `README <https://github.com/overhangio/cookiecutter-tutor-plugin/blob/master/README.rst>`__ for more information.
`Podman <https://podman.io/>`_ is a fully featured container engine that is daemonless. It provides a Docker CLI comparable command line that makes it pretty easy for people transitioning over from Docker.
Simply put, this means that you can do something like: ``alias docker=podman`` and everything will run and behave pretty much as expected.
As of podman v3.0.0, podman now officially supports ``docker-compose`` via a shim service. This means that you now have the option of running Tutor with Podman, instead of the native Docker tools.
This has some practical advantages: it does not require a running Docker daemon, and it enables you to run and build Docker images without depending on any system component running as ``root``.
..warning::
You should not attempt to run Tutor with Podman on a system that already has native ``docker`` installed. If you want to switch to ``podman`` using the aliases described here, you should uninstall (or at least stop) the native Docker daemon first.
Enabling Podman
~~~~~~~~~~~~~~~
Podman is supported on a variety of development platforms, see the `installation instructions <https://podman.io/getting-started/installation>`_ for details.
Once you have installed Podman and its dependencies on the platform of your choice, you'll need to make sure that the ``podman`` binary, usually installed as ``/usr/bin/podman``, is aliased to ``docker``.
On some CentOS and Fedora releases, you can install a package named ``podman-docker`` to do this for you, but on other platforms, you'll need to take of this yourself.
- To alias ``podman`` to ``docker``, you can simply run this command::
$ alias docker=podman
..note::
Running this command only makes a temporary alias. For a more permanent alias, you should place that command in your ``bashrc`` or equivalent file.
Getting docker-compose to work with Podman
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
To allow ``podman`` to work with ``docker-compose``, you'll need to enable a podman socket which pretends to be ``docker``.
For rootless containers, this requires you to start the ``podman.service`` as a regular user and set the ``DOCKER_HOST`` environment variable. This can be done as follows::
If you are running in rootless mode, ``tutor local`` expects a web proxy to be running on port ``80`` or port ``443``. For instructions on how to configure a web proxy, view `this tutorial <https://docs.tutor.edly.io/tutorials/proxy.html>`_.
..note::
As with the previous ``alias`` command, if you'd like to make the ``DOCKER_HOST`` variable permanent, you should put the entire export command in your ``bashrc`` or equivalent file.
Fixing SELinux Errors
~~~~~~~~~~~~~~~~~~~~~
..warning::
Disabling ``SELinux`` or setting it to *permissive mode* on your system is **highly discouraged and will render your system vulnerable.**
If your system has ``SELinux`` working in enforcing mode, chances are that the SELinux context of the tutor root directory won't be set correctly. This will cause read issues because containers will not be able read files from volumes due to a context mismatch.
Errors stemming from this will look as follows in the ``sealert`` program::
"SELinux is preventing caddy from read access on the file Caddyfile."
"SELinux is preventing celery from read access on the directory cms."
"SELinux is preventing mysqld from add_name access on the directory is_writable."
You can verify the context mismatch by running::
$ ls -lZ $(tutor config printroot)
You'll most likely see something that looks like this::
-rw-r--r--. 1 tutor tutor unconfined_u:object_r:data_home_t:s0 2145 Jan 6 20:13 config.yml
drwxr-xr-x. 2 tutor tutor unconfined_u:object_r:data_home_t:s0 6 Jan 6 20:14 data
drwxr-xr-x. 8 tutor tutor unconfined_u:object_r:data_home_t:s0 121 Jan 6 20:14 env
We're interested in the ``unconfined_u:object_r:data_home_t:s0`` part of that output.
Notice how the third part of that says ``data_home_t``?
That's the context type. For tutor to work, we need that part to be set to ``container_file_t``.
This can be done as follows::
# Set the SELinux type of the tutor root directory and all of it's subdirectories to `container_file_t`
$ sudo semanage fcontext -a -t container_file_t "$(tutor config printroot)(/.*)?"
# Apply the newly set security context to the directories
$ sudo restorecon -RF $(tutor config printroot)
Running these two commands in a sequence should fix the SELinux errors.
Verifying your environment
~~~~~~~~~~~~~~~~~~~~~~~~~~
Once you've set everything up as described, you should be able to run ``docker version`` and ``docker-compose --help`` and get a valid output.
After that, you should be able to use ``tutor local``, and other commands as if you had installed the native Docker tools.
You can then view the portainer UI at `http://localhost:9000 <http://localhost:9000>`_. You will be asked to define a password for the admin user. Then, select a "Local environment" to work on; hit "Connect" and select the "local" group to view all running containers.
..image:: ../img/portainer.png
:alt:Portainer demo
Among many other things, you'll be able to view the logs for each container, which is really useful.
In a vanilla deployment of Open edX with Tutor, a web proxy is launched to process incoming web requests. This web proxy is an instance of `Caddy <https://caddyserver.com/>`__ running inside a Docker container. This Docker container listens to ports 80 and 443 on the host.
Quite often, there is already a web proxy running on the host, and this web proxy also listens to ports 80 and 443. In such a configuration, the Caddy container will not be able to start out of the box. So you should make small changes to the Tutor configuration by running::
tutor config save --set ENABLE_WEB_PROXY=false --set CADDY_HTTP_PORT=81
With these changes, Tutor will no longer listen to ports 80 and 443 on the host. In this configuration, the Caddy container will only listen to port 81 on the host. Web requests will follow this path::
In this setup, the Caddy HTTP port (81) will be exposed to the world. Make sure to configure your server firewall to block unwanted connections to the Caddy container. Alternatively, you can configure the Caddy container to accept only local connections::
tutor config save --set ENABLE_WEB_PROXY=false --set CADDY_HTTP_PORT=127.0.0.1:81
It is then your responsibility to configure the web proxy on the host. There are too many use cases and proxy vendors, so Tutor does not provide configuration files that will work for everyone. You should configure your web proxy to:
- Capture traffic for the following hostnames: LMS_HOST, PREVIEW_LMS_HOST, CMS_HOST, as well as any additional host exposed by your plugins (MFE_HOST, ECOMMERCE_HOST, etc.). See each plugin documentation for more information.
- If SSL/TLS is enabled:
- Perform SSL/TLS termination using your own certificates.
- Forward http traffic to https.
- Set the following headers appropriately: ``X-Forwarded-Proto``, ``X-Forwarded-Port``.
- Forward all traffic to ``localhost:81`` (or whatever port indicated by CADDY_HTTP_PORT, see above).
- If possible, add support for `HTTP/3 <https://en.wikipedia.org/wiki/HTTP/3>`__, which considerably improves performance for Open edX (see `this comment <https://github.com/overhangio/tutor/issues/845#issuecomment-1566964289>`__).
..note::
If you want to run Open edX at ``https://...`` urls (as you probably do in production) it is *crucial* that the ``ENABLE_HTTPS`` flag is set to ``true``. If not, the web services will be configured to run at ``http://...`` URLs, and all sorts of trouble will happen. Therefore, make sure to continue answering ``y`` ("yes") to the quickstart dialogue question "Activate SSL/TLS certificates for HTTPS access?".
Does Open edX scale? This is the $10⁶ question when it comes to Tutor and Open edX deployments. The short answer is "yes". The longer answer is also "yes", but the details will very much depend on what we mean by "scaling".
Depending on the context, "scaling" can imply different things:
1. `Vertical scaling <https://en.wikipedia.org/wiki/Scalability#VERTICAL-SCALING>`__: increasing platform capacity by allocating more resources to a single server.
2. `Horizontal scaling <https://en.wikipedia.org/wiki/Scalability#HORIZONTAL-SCALING>`__: the ability to serve an infinitely increasing number of users with consistent performance and linear costs.
3. `High availability (HA) <https://en.wikipedia.org/wiki/High_availability>`__: the ability of the platform to remain fully functional despite one or more components being unavailable.
All of these can be achieved with Tutor and Open edX, but the method to attain either differs greatly. First of all, the range of available solutions will depend on which deployment target is used. Tutor supports installations of Open edX on a single server with the :ref:`"local" <local>` deployment target, where Docker containers are orchestrated by docker-compose. On a single server, by definition, the server is a single point of failure (`SPOF <https://en.wikipedia.org/wiki/Single_point_of_failure>`__). Thus, high availability is out of the question with a single server. To achieve high availability, it is necessary to deploy to a cluster of multiple servers. But while docker-compose is a great tool for managing single-server deployments, it is simply inappropriate for deploying to a cluster. Tutor also supports deploying to a Kubernetes cluster (see :ref:`k8s`). This is the recommended solution to deploy Open edX "at scale".
Scaling with a single server
----------------------------
Options are limited when it comes to scaling an Open edX platform deployed on a single-server. High availability is out of the question and the number of users that your platform can serve simultaneously will be limited by the server capacity.
Fortunately, Open edX was designed to run at scale -- most notably at `edX.org <edx.org>`__, but also on large national education platforms. Thus, performance will not be limited by the backend software, but only by the hardware.
Increasing web server capacity
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
As the server CPU and memory are increased, the request throughput can be increased by adjusting the number of uWSGI workers (see :ref:`configuration docs <openedx_configuration>`). By default, the "lms" and "cms" containers each spawn 2 uWSGI workers. The number of workers should be increased if you observe an increase in the latency of user requests but CPU usage remains below 100%. To increase the number of workers for the LMS and the CMS, run for example::
tutor config save \
--set OPENEDX_LMS_UWSGI_WORKERS=8 \
--set OPENEDX_CMS_UWSGI_WORKERS=4
tutor local restart lms cms
The right values will very much depend on your server's available memory and CPU performance, as well as the maximum number of simultaneous users who use your platform. As an example data point, it was reported that a large Open edX platform can serve up to 500k unique users per week on a virtual server with 8 vCPU and 16 GB memory.
Offloading data storage
~~~~~~~~~~~~~~~~~~~~~~~
Aside from web workers, the most resource-intensive services are in the data persistence layer. They are, by decreasing resource usage:
- `Elasticsearch <https://www.elastic.co/elasticsearch/>`__: indexing of course contents and forum topics, mostly for search. Elasticsearch is never a source of truth in Open edX, and the data can thus be trashed and re-created safely.
- `MySQL <https://www.mysql.com>`__: structured, consistent data storage which is the default destination of all data.
- `MongoDB <https://www.mongodb.com>`__: structured storage of course data.
- `Redis <https://redis.io/>`__: caching and asynchronous task management.
- `MinIO <https://min.io>`__: S3-like object storage for user-uploaded files, which is enabled by the `tutor-minio <https://github.com/overhangio/tutor-minio>`__ plugin. It is possible to replace MinIO by direct filesystem storage (the default), but scaling will then become much more difficult down the road.
When attempting to scale a single-server deployment, we recommend starting by offloading some of these stateful data storage components, in the same order of priority. There are multiple benefits:
1. It will free up some resources both for the web workers and the data storage components.
2. It is the first step towards horizontal scaling of the web workers.
3. It becomes possible to either install every component as a separate service or rely on 3rd-party SaaS with high availability.
Moving each of the data storage components is a fairly straightforward process, although details vary for every component. For instance, for the MySQL database, start by disabling the locally running MySQL instance::
tutor config save --set RUN_MYSQL=false
Then, migrate the data located at ``$(tutor config printroot)/data/mysql`` to the new MySQL instance. Configure the Open edX platform to point at the new database::
tutor config save \
--set MYSQL_HOST=yourdb.com \
--set MYSQL_PORT=3306 \
--set MYSQL_ROOT_USERNAME=root \
--set MYSQL_ROOT_PASSWORD=p4ssw0rd
The changes will be taken into account the next time the platform is restarted.
Beware that moving the data components to dedicated servers has the potential of creating new single points of failure (`SPOF <https://en.wikipedia.org/wiki/Single_point_of_failure>`__). To avoid this situation, each component should be installed as a highly available service (or as a highly available SaaS).
Scaling with multiple servers
-----------------------------
Horizontally scaling web services
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
As the number of users of a web platform increases, they put increased pressure on the web workers that respond to their requests. Thus, in most cases, web worker performance is the first bottleneck that system administrators have to face when their service becomes more popular. Initially, any given Kubernetes-based Tutor platform ships with one replica for each deployment. To increase (or reduce) the number of replicas for any given service, run ``tutor k8s scale <name> <number of replicas>``. Behind the scenes, this command will trigger a ``kubectl scale --replicas=...`` command that will seamlessly increase the number of pods for that deployment.
In Open edX multiple web services are exposed to the outside world. The ones that usually receive the most traffic are, in decreasing order, the LMS, the CMS, and the forum (assuming the `tutor-forum <https://github.com/overhangio/tutor-forum>`__ plugin was enabled). As an example, all three deployment replicas can be scaled by running::
tutor k8s scale lms 8
tutor k8s scale cms 4
tutor k8s scale forum 2
Highly-available architecture, autoscaling, ...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
There is only so much that Tutor can do for you, and scaling some components falls beyond the scope of Tutor. For instance, it is your responsibility to make sure that your Kubernetes cluster has a `highly available control plane <https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/high-availability/>`__ and `topology <https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/ha-topology/>`__. Also, it is possible to achieve `autoscaling <https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/>`__; but it is your responsibility to setup latency metrics collection and to configure the scaling policies.
Comprehensive theming is enabled by default, but only the default theme is compiled. `Indigo <https://github.com/overhangio/indigo>`__ is a better, ready-to-run theme that you can start using today.
To compile your own theme, add it to the ``env/build/openedx/themes/`` folder::
The ``themes`` folder should have the following structure::
openedx/themes/
mycustomtheme1/
cms/
...
lms/
...
mycustomtheme2/
...
Then you must rebuild the openedx Docker image::
tutor images build openedx
Finally, you should enable your theme with the :ref:`settheme command <settheme>`.
.._theme_development:
Developing a new theme
----------------------
With Tutor, it's pretty easy to develop your own themes. Start by placing your files inside the ``env/build/openedx/themes`` directory. For instance, you could start from the ``edx.org`` theme present inside the ``edx-platform`` repository::
You should not create a soft link here. If you do, it will trigger a ``Theme not found in any of the themes dirs`` error. This is because soft links are not properly resolved from inside docker containers.
Then, run a local webserver::
tutor dev start lms
The LMS can then be accessed at http://local.edly.io:8000. You will then have to :ref:`enable that theme <settheme>`::
tutor dev do settheme mythemename
Watch the themes folders for changes (in a different terminal)::
tutor dev run watchthemes
Make changes to some of the files inside the theme directory: the theme assets should be automatically recompiled and visible at http://local.edly.io:8000.
You have gone through the :ref:`Quickstart installation <quickstart>`: at this point you should have a running Open edX platform. If you don't, please follow the instructions from the :ref:`Troubleshooting <troubleshooting>` section.
You have gone through the :ref:`Quickstart installation <quickstart>`: at this point, you should have a running Open edX platform. If you don't, please follow the instructions from the :ref:`Troubleshooting <troubleshooting>` section.
Logging-in as administrator
---------------------------
Out of the box, Tutor does not create any user for you. You will want to create a user yourself with staff and administrator privileges in order to access the studio. There is a :ref:`simple command for that <createuser>`.
Out of the box, Tutor does not create any user for you. You will want to create a user yourself with staff and administrator privileges to access the studio. There is a :ref:`simple command for that <createuser>`.
Importing a demo course
-----------------------
To get a glimpse of the possibilities of Open edX, we recommend you import the `official demo test course <https://github.com/edx/demo-test-course>`__. Tutor provides a :ref:`simple command for that <democourse>`.
To get a glimpse of the possibilities of Open edX, we recommend you import the `official demo test course <https://github.com/openedx/edx-demo-course>`__. Tutor provides a :ref:`simple command for that <democourse>`.
Making Open edX look better
---------------------------
Tutor makes it easy to :ref:`develop <theming>` and :ref:`install <settheme>` your own themes. We also provide `Indigo <https://github.com/overhangio/indigo>`__: a free, customizable theme that you can install today.
Tutor makes it easy to :ref:`install <theming>` and :ref:`develop <theme_development>` your own themes. We also provide `Indigo <https://github.com/overhangio/indigo>`__: a free, customizable theme that you can install today.
Adding features
---------------
@ -35,7 +35,12 @@ Deploying to Kubernetes
Yes, Tutor comes with Kubernetes deployment support :ref:`out of the box <k8s>`.
Gathering insights and analytics about Open edX
-----------------------------------------------
Check out `Cairn <https://github.com/overhangio/tutor-cairn>`__, the next-generation analytics solution for Open edX.
Meeting the community
---------------------
Ask your questions and chat with the Tutor community on the official community forums: https://discuss.overhang.io
Ask your questions and chat with the Tutor community on the official Open edX community forum: https://discuss.openedx.org
Régis Behmo
Syed Muhammad Dawoud Sheraz Ali
Abdul-Muqadim-Arbisoft
Alejandro Cardenas
Danyal Faheem
Overhang.IO
Emad Rad
Omar Al-Ithawi
Florian Haas
Talha Rizwan
Kyle McCormick
Fateme Khodayari
David Ormsbee
Paulo Viadanna
Pablo Thasi
dependabot[bot]
0x29a
Sarina Canelake
Braden MacDonald
Emad Rad
Johan Castiblanco
Brian Mesick
Kyle McCormick
Kyle McCormick
Moisés González
Maria Fernanda Magallanes Zubillaga
Maria Grimaldi
Keith Grootboom
Carlos Muniz
Abhiram Kuchibhotla
Abderraouf Mehdi Bouhali
Daniel Herrmann
james
Adolfo R. Brandes
Ghassan Maslamani
Michael Wheeler
Frank Anderson
Carlos Muniz
Sofiane Bebert
David Ormsbee
Ed D'Azzo
Prakash Divy
Foad Lind
Gábor Boros
iamcristye@outlook.com
ibrmora
Andrés González
Navin Karkera
Daniel Rudnitski
AmiT
Erick Machado
Matjaz Gregoric
Florian Haas
Rebecca Graber
Jayram Nai
Ned Batchelder
alex.soh
Derek Crosson
Maxim Beder
Peter Parkkali
Kevin Valencia
Shimul Chowdhury
Michael Wheeler
ChrisChV
