This change builds upon a previously proposed PR:
https://github.com/overhangio/tutor/pull/437
There was another long conversation about this topic here:
https://github.com/overhangio/tutor-forum/pull/10#issuecomment-1314799915
We could have supported the MongoDB auth/replica set/ssl parameters as part of
the MongoDB host URI, but then this URI is not supported in the forum plugin,
which uses an old version of the mongoid client. We were hoping that the client
would have been upgraded by now, but it's not been upgraded for a long time.
The changes introduced here are 100% backward-compatible. The forum plugin will
have to be updated to take into account the new parameters.
The following syntax is only supported in jinja2>=2.10:
{% set jwt_rsa_key | rsa_import_key %}{{ JWT_RSA_PRIVATE_KEY }}{% endset %}
Thus, we bump the minimal working version of jinja2 in the base requirements.
See discussion: https://discuss.openedx.org/t/error-while-tutor-local-quickstart/8796
- [Improvement] Auto-completion of `plugins` and `config` arguments: `plugins enable/disable NAME`, `plugins install PATH`, `config save --set KEY=VAL`, `config save --unset KEY`, `config printvalue KEY`. (by @regisb)
- [Bugfix] Fix minimum click version (>= 8.0.0) when installing tutor from pip.
- [Improvement] Enable CORS by default for both LMS and CMS by moving those settings to the `common_all` partial. (by @arbrandes)
Changelog management was starting to be a hassle:
- there were conflicts every time a PR was merged
- there were conflicts every time we merged the nightly branch in the new
release branch, or vice versa.
Now, all changelog entries are stored as separate files in changelog.d,
including nightly. Nightly entries will be collected for every major release.
The ENABLE_CORS_HEADERS feature flag is already true for the LMS.
Instead of duplicating it for Studio via yaml settings, make this a
common Django setting to both LMS and CMS and all their environments.
- [Improvement] Auto-complete implicit `local/dev --mount /path/to/...` options. (by @regisb)
- 💥[Feature] Strong typing of action and filter hooks: this allows us to detect incorrect calls to `actions.add` or `filters.add` early. Strong typing forces us to break the `do` and `apply` API by removing the `context` named argument. Developers should replace `do(context=...)` by `do_from_context(..., )` (and similar for `apply`).
When typing `tutor local run --mount /path/to/edx-pl<TAB>`, the mount option
should be auto-completed to the full edx-platform repo path. That is, if shell
completion is enabled:
https://docs.tutor.overhang.io/install.html#shell-autocompletion
Here, we make sure that the implicit form of the `--mount` argument is properly
auto-completed. We are unable to get completion to work in the explicit form,
because args that include colons do not even reach the `shell_completion`
method.
Now that the mypy bugs have been resolved, we are able to define more precisely
and cleanly the types of Actions and Filters.
Moreover, can now strongly type named actions and hooks (in consts.py). With
such a strong typing, we get early alerts of hooks called with incorrect
arguments, which is nothing short of awesome :)
This change breaks the hooks API by removing the `context=...` argument. The
reason for that is that we cannot insert arbitrary arguments between `P.args,
P.kwargs`: https://peps.python.org/pep-0612/#the-components-of-a-paramspec
> A function declared as def inner(a: A, b: B, *args: P.args, **kwargs:
> P.kwargs) -> R has type Callable[Concatenate[A, B, P], R]. Placing
> keyword-only parameters between the *args and **kwargs is forbidden.
Getting the documentation to build in nitpicky mode is quite difficult... We
need to add `nitpick_ignore` to the docs conf.py, otherwise sphinx complains
about many missing class references. This, despite upgrading almost all doc
requirements (except docutils).
- [Security] Fix xblock ajax handler vulnerability. (by @regisb)
- [Improvement] Use web proxy gzip encoding to improve bandwidth. We
observe a 75% size reduction on the LMS dashboard. (by @ghassanmas)
Compressing assests would lead to readuce transfer size.
As testing with frontend-app-learning/Olive, the network traffic
before was about ~4MB, after this it became ~1MB.
This change was suggested by Google Lighthouse[1], there are of
course more suggestion but this was one the easiest and one of most
impactful.
Also check orignal PR overhangio/tutor-mfe/pull/64 for more
info.
[1]: https://web.dev/uses-text-compression
- [Improvement] Upgrade Scorm XBlock to v14.0.0. (by @regisb)
- 💥[Improvement] The Richie plugin was transferred to the Openfun organization; thus, it is no longer officially supported and it is removed from the default set of plugins that ships with
`pip install tutor[full]` or the Tutor pre-compiled binary. Users are encouraged to uninstall the `tutor-richie` Python package and install the `tutor-contrib-richie` package instead.
- [Feature] Upgrade edx-platform i18n strings to nutmeg.2. (by @regisb)
Strings could not be pulled from transifex because the file names were
incorrect. This is now fixed and we are now able to pull the i18n strings from
the nutmeg.2 tag.
The pymongo dependency for edx-platform was updated (3.10.1 to 3.12.3)
in https://github.com/openedx/edx-platform/pull/30569
This caused the following error when running the edx-platform database
migration split_modulestore_django.0002_data_migration as part of
`tutor dev quickstart`:
pymongo.errors.ServerSelectionTimeoutError: client is configured to
connect to a replica set named '' but this node belongs to a set named
'None', Timeout: 30s, Topology Description: <TopologyDescription id:
62bdbaf182687350acf1aeec, topology_type: Single, servers:
[<ServerDescription ('mongodb', 27017) server_type: Unknown, rtt:
None, error=ConfigurationError("client is configured to connect to a
replica set named '' but this node belongs to a set named 'None'")>]>
This commit explicitly sets replicaSet to None to indicate that it's a
standalone MongoDB instance. I also had to remove the CONTENTSTORE entry
from auth.yml because edx-platform's devstack.py assumes it has a
non-null value (set in common.py), and devstack.py executes before
tutor's development.py can set this replicaSet value.
- [Feature] Add the `-m/--mount` option to `tutor dev quickstart`.
- [Bugfix] Fix `tutor dev start -m /path/to/frontend-app-learning` by introducing dev-specific `COMPOSE_DEV_TMP` and `COMPOSE_DEV_JOBS_TMP` filters (by @regisb).
- [Bugfix] Log the shell commands that Tutor executes more accurately. (by @kdmccormick)
- [Bugfix] `tutor dev quickstart` would fail under certain versions of docker-compose due to a bug in the logic that handled volume mounting. (by @kdmccormick)
- [Bugfix] The `tutor k8s start` command will succeed even when `k8s-override` and `kustomization-patches-strategic-merge` are not specified. (by @edazzocaisser)
- [BugFix] `kubectl wait` checks deployments instead of pods as it could hang indefinitely if there are extra pods in a broken state. (by @keithgg)
The -m/--mount option makes it possible to bind-mount volumes at runtime. The
volumes are declared in a local/docker-compose.tmp.yml file. The problem with
this approach is when we want to bind-mount a volume to a service which is
specific to the dev context. For instance: the "learning" service when the MFE
plugin is enabled.
In such a case, starting the service triggers a call to `docker-compose stop`
in the local context. This call fails because the "learning" service does not
exist in the local context. Note that this issue only seems to occur with
docker-compose v1.
To resolve this issue, we create two additional filters for
the dev context, which emulate the behaviour of the local context. With this approach, we convert the -m/--mount arguments right after they are parsed. Because they are parsed just once, we can get rid of the de-duplication logic initially introduced with the COMPOSE_CLI_MOUNTS context.
Close #711. Close also https://github.com/overhangio/tutor-mfe/issues/57.
Whenever Tutor executes a shell command, it logs out said
command in order to aid in end user understanding/debugging.
In some cases (notably, when running jobs in containers)
the logged command was not accurately quoted. The command
was run correctly, because it was passed in pieces to
``subprocess.Popen``, which correctly joins the pieces together
into a valid POSIX shell command; however, the logged version
of the command was constructed by simply joining the pieces
with spaces. This usually works, but breaks down when running
complex shell commands with nested quoting.
This commit changes the logging to use ``shlex.join``, which
joins command pieces together in a POSIX-compliant way,
presumably the same way as ``subprocess.Popen``.
Example:
tutor local importdemocourse
runs the shell command:
docker-compose -f /home/kyle/.local/share/tutor/env/local/docker-compose.yml -f /home/kyle/.local/share/tutor/env/local/docker-compose.prod.yml -f /home/kyle/.local/share/tutor/env/local/docker-compose.tmp.yml --project-name tutor_local -f /home/kyle/.local/share/tutor/env/local/docker-compose.jobs.yml -f /home/kyle/.local/share/tutor/env/local/docker-compose.jobs.tmp.yml run --rm cms-job sh -e -c 'echo "Loading settings $DJANGO_SE... (several more script lines) ...eindex_course --all --setup'
but the logged shell command was:
docker-compose -f /home/kyle/.local/share/tutor/env/local/docker-compose.yml -f /home/kyle/.local/share/tutor/env/local/docker-compose.prod.yml -f /home/kyle/.local/share/tutor/env/local/docker-compose.tmp.yml --project-name tutor_local -f /home/kyle/.local/share/tutor/env/local/docker-compose.jobs.yml -f /home/kyle/.local/share/tutor/env/local/docker-compose.jobs.tmp.yml run --rm cms-job sh -e -c echo "Loading settings $DJANGO_SE... (several more script lines) ...eindex_course --all --setup
which will not run if copied and pasted back into the
user's terminal, as the importdemocourse shell script is unquoted.
When waiting for pods, it's possible that the deployment may be
complete but, because other pods may have been Evicted or Killed, the
wait wait condition completes.
In certain code paths, such as in `tutor local quickstart`,
`process_mount_points` is called more than once in the same process,
causing mounts to be added to `COMPOSE_LOCAL[_JOBS]_TMP` redundantly.
As a result, docker-compose[.jobs].tmp.yml was occasionally being
rendered with duplicate volume specifiers. Some versions of Docker
Compose ignored this; other versions warned or threw an error.
In order to make `process_mount_points` tolerant to being called
multiple times, we wrap its volume-adding callbacks within a new
hooks context. This allows us to clear said hooks context every
time `process_mount_points` is called, essentially making the
function idempotent.
Co-authored-by: Régis Behmo <regis@behmo.com>
- [Bugfix] Build openedx-dev Docker image even when the host user is root, for instance on Windows. (by @regisb)
- [Bugfix] Patch nutmeg.1 release with [LTI 1.3 fix](https://github.com/openedx/edx-platform/pull/30716). (by @ormsbee)
- [Improvement] Make it possible to override k8s resources in plugins using `k8s-override` patch. (by @foadlind)
