Templated hooks we almost completely useless, so we get rid of them.
This allows us to get rid entirely of hook names and hook indexes, which
makes the whole implementation much simpler. Hook removal (with
`clear_all`) is achieved thanks to weak references.
This is an important change, where we get remove the previous `--mount`
option, and instead opt for persistent bind-mounts.
Persistent bind mounts have several advantages:
- They make it easier to remember which folders need to be bind-mounted.
- Code is *much* less clunky, as we no longer need to generate temporary
docker-compose files.
- They allow us to bind-mount host directories *at build time* using the
buildx `--build-context` option.
- The transition from development to production becomes much easier, as
images will automatically be built using the host repo.
The only drawback is that persistent bind-mounts are slightly less
portable: when a config.yml file is moved to a different folder, many
things will break if the repo is not checked out in the same path.
For instance, this is how to start working on a local fork of
edx-platform:
tutor config save --append MOUNTS=/path/to/edx-platform
And that's all there is to it. No, this fork will be used whenever we
run:
tutor images build openedx
tutor local start
tutor dev start
This change is made possible by huge improvements in the build time
performance. These improvements make it convenient to re-build Docker
images often.
Related issues:
https://github.com/openedx/wg-developer-experience/issues/71https://github.com/openedx/wg-developer-experience/issues/66https://github.com/openedx/wg-developer-experience/issues/166
It was useless to create a *-permissions job for every application.
Instead, we create a single "permissions" service. It can be extended
via the "docker-compose-permissions-command" patch.
Instead, the compose plugin must be installed.
We deprecate docker-compose because v1 is not supported starting from
the end of June 2023.
See "evolution of compose": https://docs.docker.com/compose/compose-v2/
Among other changes: ORA2 file uploads were stored in a folder named
"SET-ME-PLEASE (ex. bucket-name)" (sigh). With this change, the folder
should be automatically renamed to "openedxuploads". This issue has been
occuring since June 2019... (sigh²)
Close #707
in some cases, tutor might run inside a pod, which that pod has access to a cluster via role binding and a service account. this way, there's no ./kube/config file, but kubectl commands run with no issue.
Close #843
in some cases, tutor might run inside a pod, which that pod has access to a cluster via role binding and a service account. this way, there's no ./kube/config file, but kubectl commands run with no issue.
Close #843
This paves the way for `docker buildx build` and better caching.
For instance, with this change you can try out the following plugin,
which should make image building much faster in CI:
https://gist.github.com/regisb/4049622ec4b48cbd48c89ec708dc5252
(not ready for production just yet, we still need to build and push the
images)
In the output of `plugins list` it's difficult to see which plugins are
enabled at a glance. This change adds a more visible checkmark to the
output.
Ex:
$ tutor plugins list
NAME STATUS VERSION
cairn ✅ enabled 15.0.3
discovery installed 15.0.0
ecommerce installed 15.0.1
forum installed 14.0.0
mfe ✅ enabled 15.0.5
minio installed 15.1.0
Before this commit, setting up an edx-platform development environment
took multiple steps:
tutor dev launch
tutor dev run --mount=/path/to/edx-platform lms bash
>> pip install -e .
>> npm clean-install
>> openedx-assets build --env=dev
This commit moves the steps under ``run`` into an init task, which
is automatically run by ``launch``. Thus, setup is now one command:
tutor dev launch --mount=edx-platform
These extra init steps are only applicable when bind-mounting
edx-platform (because bind-mounting the repository overrides
some important artifacts that exist on the image, which must be
re-generated). Thus, the new init tasks exists early if it detects
that it is *not* operating on a bind-mounted repository.
Finally, we try to simplify the Open edX development docs so that
it is clearer how bind-mounting fits into the development process.
These bind-mounts:
* ../build/openedx/themes:/openedx/themes
* ../build/openedx/requirements:/openedx/requirements
existed in the dev lms and cms containers, but they did
not exist in the lms-job and cms-job containers.
This means that themes and requirements that were *built into the
image* would exist in the job containers, but live updates to the
themes and requirements would not apply.
To resolve this, we set ``volumes:`` on the lms-job and cms-job
services so that they match the volumes for the normal lms and
cms services.
Part of: https://github.com/openedx/wg-developer-experience/issues/146
Closes: https://github.com/openedx/wg-developer-experience/issues/152
This works around (but does not close) these related issues:
* https://github.com/openedx/wg-developer-experience/issues/150
* https://github.com/openedx/wg-developer-experience/issues/151
The lms-job and cms-job services were configured to use
{{ DOCKER_IMAGE_OPENEDX }} rather than {{ DOCKER_IMAGE_OPENEDX_DEV }}.
This means that when running jobs in dev mode, a la:
tutor dev do init
a production image would be used, to the user's surprise.
When a job is invoked, we now replace the job in k8s/jobs.yml
instead of rewriting jobs.yml to only contain the relevant
job. This allows patchStrategicMerge to work for jobs.
We implement this TEP: https://discuss.openedx.org/t/tutor-enhancement-proposal-tep-plugin-indices/8182
With plugin indexes, tutor users can install and upgrade plugins directly from indexes:
tutor plugins install ecommerce
tutor plugins index add contrib
tutor plugins install codejail
tutor plugins upgrade all
This change has been long in the coming \o/
Users want to be able to override the request `max_size` to upload
larger files. But they will not be able to if the patch is placed after
the `request` directive. So we move the patch statement before the
directive. Also, we wrap the `request_body` directives within `handle`
statements. If not, then different sizes are not managed properly.
To override the max upload size in the cms, add the following to the
"caddyfile-cms" patch:
handle_path /import/* {
request_body {
max_size 500MB
}
}
See discussion:
https://discuss.openedx.org/t/how-to-update-caddyfile-using-tutor-plugin/8944
The hooks API had several issues which are summarized in this comment:
https://github.com/openedx/wg-developer-experience/issues/125#issuecomment-1313553526
1. "consts" was a bad name
2. "hooks.filters" and "hooks.Filters" could easily be confused
3. docs made it difficult to understand that plugin developers should use the catalog
To address these issues, we:
1. move "consts.py" to "catalog.py"
2. Remove "hooks.actions", "hooks.filters", "hooks.contexts" from the API.
3. re-organize the docs and give better usage examples in the catalog.
This change is a partial fix for https://github.com/openedx/wg-developer-experience/issues/125
Adds `from __future__ import annotations` to the top of every module,
right below the module's docstring. Replaces any usages of t.List,
t.Dict, t.Set, t.Tuple, and t.Type with their built-in equivalents:
list, dict, set, tuple, and type. Ensures that make test still passes
under Python 3.7, 3.8 and 3.9.
The LMS and CMS were producing lots of logs similar to:
cms_1 | 2023-01-17 15:30:11,359 INFO 7 [openedx.core.djangoapps.cors_csrf.helpers] [user 7] [ip 31.223.46.44] helpers.py:64 - Origin 'https://studio.demo.openedx.overhang.io' was not in `CORS_ORIGIN_WHITELIST`; full referer was 'https://studio.demo.openedx.overhang.io/learning/course/course-v1:edX+DemoX+Demo_Course/home' and requested host was 'studio.demo.openedx.overhang.io'; CORS_ORIGIN_ALLOW_ALL=False
These warnings are produced by openedx.core.djangoapps.cors_csrf.helpers. I
don't think they indicate any problem, but they pollute the logs. They are
resolved by adding the "http(s)://<lms/cms host>" to CORS_ORIGIN_WHITELIST in
production, so we did just that.
When a user registers, they receive a confirmation email. This email contained
two links to "https://example.com/..." urls. This was caused by the fact that
the default site, indicated by SITE_ID=1, was example.com. We resolve this
issue by setting instead SITE_ID=2, which should point to the site with the LMS
domain name.
This is a potentially breaking change for platforms that have manually set to 1
the id of the LMS site in the database. These platforms should now set
SITE_ID=1 via a plugin.
Alternatives we have considered include modifying the id field of the LMS site
in the database. Unfortunately such a change would have important consequences,
as the site ID is used as a foreign key for other models.
Note that non-https sites still include https links in the registration emails.
This is because the "https" scheme is hardcoded by the "ensure_url_is_absolute"
utility function. So there is nothing we can do about this without making
changes upstream.
Close #572.
Adds `from __future__ import annotations` to the top of every module,
right below the module's docstring. Replaces any usages of t.List,
t.Dict, t.Set, t.Tuple, and t.Type with their built-in equivalents:
list, dict, set, tuple, and type. Ensures that make test still passes
under Python 3.7, 3.8 and 3.9.
This removes an openedx/edx-platform commit backported as a patch to tutor to olive.1 release
Since the commit is already merged into edx-platform:master branch used
by tutor nightly, there is no further need for it.
- [Improvement] Upgrade ipdb and ipython packages in the openedx development image. (by @regisb)
- [Improvement] Skip unnecessary image building in development. This should make `tutor dev launch` slightly faster. (by @regisb)
- [Bugfix] Fix Authn MFE login in development by disabling enterprise integration. (by @regisb)
- [Bugfix] Fix "Invalid value for ‘--from’" when running `tutor local upgrade --from=nutmeg`. If you are facing this error, just run `tutor local launch` and your platform should be automatically upgraded.
- [Bugfix] Fix "TypeError: Parameters to Generic[...] must all be type variables" error. This error may occur when upgrading from a very old installation of Tutor. It is due to an old version of the typing-extensions package.
- 💥[Deprecation] Get rid of the `quickstart` command. v15.0.0 introduced a deprecation warning, but we actually want users to stop using this command. Instead, use `launch` (by @regisb).
- [Improvement] Backfill persistent grades during upgrade from Nutmeg. If you observe missing grades after the upgrade from Nutmeg, run `tutor local upgrade --from=nutmeg`. (by @regisb)
In development, login via the authn mfe was broken because of explicit
enterprise integration:
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=18000): Max retries exceeded with url: /enterprise/api/v1/enterprise-learner/?username=regis (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fd4c02b8a90>: Failed to establish a new connection: [Errno 111] Connection refused'))
See: https://discuss.overhang.io/t/tutor-login-fail-in-new-version/3083
I have no idea why we decided to kickstart a separate build. The image will be
built anyway at the next step because we run `docker compose up --build` in
`tutor dev start`.
The build step was introduced in this PR: https://github.com/overhangio/tutor/pull/627
- [Bugfix] Fix `jinja2.exceptions.TemplateSyntaxError: expected token 'end of statement block', got '|'` error by bumping the minimum required version of the Jinja2 package.
- [Feature] Add support for MongoDB SSL, authentication source, mechanism and replica set via the `MONGODB_USE_SSL`, `MONGODB_AUTH_MECHANISM`, `MONGODB_AUTH_SOURCE`, `MONGODB_REPLICA_SET` settings. (by @zakum1 and @regisb)
- [Bugfix] Fix tag of "openedx" development Docker image. Previously, this Docker tag did not include the Tutor version. As a consequence, a different cached image could be used in some cases. For instance: when running `tutor dev run` commands. Now, the image tag is "openedx-dev:TUTOR_VERSION".
- [Bugfix] Fix name of Swahili locale: it is "sw-ke" and not "sw" (by @regisb).
- [Security] Apply drag-n-drop v2 xblock [security patch](https://discuss.openedx.org/t/upcoming-security-release-xblock-drag-and-drop-v2/8768/7). (by @regisb)
When running:
tutor dev run -m /path/to/edx-platform lms
pip install -r requirements/edx/development.txt
I realised that I was re-installing packages that should already have been
present in the image. The reason for that was that I was running an outdated
version of the dev version of the openedx Docker image. This happens because
`tutor dev run` does not trigger an image re-build.
We solve this issue by pinning the openedx dev Docker image tag to the current
tutor version.
This change builds upon a previously proposed PR:
https://github.com/overhangio/tutor/pull/437
There was another long conversation about this topic here:
https://github.com/overhangio/tutor-forum/pull/10#issuecomment-1314799915
We could have supported the MongoDB auth/replica set/ssl parameters as part of
the MongoDB host URI, but then this URI is not supported in the forum plugin,
which uses an old version of the mongoid client. We were hoping that the client
would have been upgraded by now, but it's not been upgraded for a long time.
The changes introduced here are 100% backward-compatible. The forum plugin will
have to be updated to take into account the new parameters.
- [Improvement] Auto-completion of `plugins` and `config` arguments: `plugins enable/disable NAME`, `plugins install PATH`, `config save --set KEY=VAL`, `config save --unset KEY`, `config printvalue KEY`. (by @regisb)
- [Bugfix] Fix minimum click version (>= 8.0.0) when installing tutor from pip.
- [Improvement] Enable CORS by default for both LMS and CMS by moving those settings to the `common_all` partial. (by @arbrandes)
The ENABLE_CORS_HEADERS feature flag is already true for the LMS.
Instead of duplicating it for Studio via yaml settings, make this a
common Django setting to both LMS and CMS and all their environments.
- [Improvement] Auto-complete implicit `local/dev --mount /path/to/...` options. (by @regisb)
- 💥[Feature] Strong typing of action and filter hooks: this allows us to detect incorrect calls to `actions.add` or `filters.add` early. Strong typing forces us to break the `do` and `apply` API by removing the `context` named argument. Developers should replace `do(context=...)` by `do_from_context(..., )` (and similar for `apply`).
When typing `tutor local run --mount /path/to/edx-pl<TAB>`, the mount option
should be auto-completed to the full edx-platform repo path. That is, if shell
completion is enabled:
https://docs.tutor.overhang.io/install.html#shell-autocompletion
Here, we make sure that the implicit form of the `--mount` argument is properly
auto-completed. We are unable to get completion to work in the explicit form,
because args that include colons do not even reach the `shell_completion`
method.
Now that the mypy bugs have been resolved, we are able to define more precisely
and cleanly the types of Actions and Filters.
Moreover, can now strongly type named actions and hooks (in consts.py). With
such a strong typing, we get early alerts of hooks called with incorrect
arguments, which is nothing short of awesome :)
This change breaks the hooks API by removing the `context=...` argument. The
reason for that is that we cannot insert arbitrary arguments between `P.args,
P.kwargs`: https://peps.python.org/pep-0612/#the-components-of-a-paramspec
> A function declared as def inner(a: A, b: B, *args: P.args, **kwargs:
> P.kwargs) -> R has type Callable[Concatenate[A, B, P], R]. Placing
> keyword-only parameters between the *args and **kwargs is forbidden.
Getting the documentation to build in nitpicky mode is quite difficult... We
need to add `nitpick_ignore` to the docs conf.py, otherwise sphinx complains
about many missing class references. This, despite upgrading almost all doc
requirements (except docutils).
We introduce a new filter to implement custom commands in arbitrary containers.
It becomes easy to write convenient ad-hoc commands that users will
then be able to run either on Kubernetes or locally using a documented CLI.
Pluggable jobs are declared as Click commands and are responsible for
parsing their own arguments. See the new CLI_DO_COMMANDS filter.
Close https://github.com/overhangio/2u-tutor-adoption/issues/75
Nothing revolutionary here, we just implement the same priority queue that
existed in actions. It will be necessary to trigger init tasks in the right
order.
- [Security] Fix xblock ajax handler vulnerability. (by @regisb)
- [Improvement] Use web proxy gzip encoding to improve bandwidth. We
observe a 75% size reduction on the LMS dashboard. (by @ghassanmas)
Compressing assests would lead to readuce transfer size.
As testing with frontend-app-learning/Olive, the network traffic
before was about ~4MB, after this it became ~1MB.
This change was suggested by Google Lighthouse[1], there are of
course more suggestion but this was one the easiest and one of most
impactful.
Also check orignal PR overhangio/tutor-mfe/pull/64 for more
info.
[1]: https://web.dev/uses-text-compression
Régis Behmo
Overhang.IO
Florian Haas
Emad Rad
Emad Rad
Johan Castiblanco
Kyle McCormick
Kyle McCormick
Moisés González
Maria Fernanda Magallanes Zubillaga
Maria Grimaldi
Keith Grootboom
Carlos Muniz
Abderraouf Mehdi Bouhali
james
Adolfo R. Brandes
Ghassan Maslamani