The `compilejsi18n` command was failing during image building because the
Open-edX package was not installed properly. The reason for that was an earlier
change where we got rid of the `pip install -r requirements/edx/local.in`
command. Installing the Open-edX package was part of this requirement file.
The local.in requirements file no longer exists, but we still need to `pip
install -e .` the edx-platform repo. To run this command we need both the
edx-platform repo and the virtualenv.
The good news is that there are no more local requirements in the base.txt
requirements file. This means that we no longer have to COPY the edx-platform
repo in the requirements installation step. Thus, changes in edx-platform will
no longer trigger a rebuild of the pip requirements; this means that re-builds
will be much faster when making changes to edx-platform.
Note that plugins that implemented the
"openedx-dockerfile-post-python-requirements" patch and that needed access to
the edx-platform repo will no longer work. Instead, these plugins should
implement the "openedx-dockerfile-pre-assets" patch. This scenario should be
very rare, though.
Close #726
The local requirements files does not exist since local requirements were all
removed from the edx-platform repo. As a consequence, the nightly build was
broken.
`quickstart` is being renamed to `launch` and deprecated in favor of
using `launch`. The `quickstart` function temporarily aliases to
`launch`. Further mentions of `quickstart` have been changed to
reference `launch` instead.
We are indicating that this change is breaking 💥 to encourage people to
migrate their scripts right away!
Strings could not be pulled from transifex because the file names were
incorrect. This is now fixed and we are now able to pull the i18n strings from
the nutmeg.2 tag.
Soon, running:
pip install -r ./requirements/edx/base.txt
in edx-platform will no longer install the local
project (that is, `-e .`). To prepare for that change,
we add the line:
pip install -e .
to the Dockerfile. This is backwards-compatible.
More details:
https://openedx.atlassian.net/browse/BOM-2575?focusedCommentId=613181
The pymongo dependency for edx-platform was updated (3.10.1 to 3.12.3)
in https://github.com/openedx/edx-platform/pull/30569
This caused the following error when running the edx-platform database
migration split_modulestore_django.0002_data_migration as part of
`tutor dev quickstart`:
pymongo.errors.ServerSelectionTimeoutError: client is configured to
connect to a replica set named '' but this node belongs to a set named
'None', Timeout: 30s, Topology Description: <TopologyDescription id:
62bdbaf182687350acf1aeec, topology_type: Single, servers:
[<ServerDescription ('mongodb', 27017) server_type: Unknown, rtt:
None, error=ConfigurationError("client is configured to connect to a
replica set named '' but this node belongs to a set named 'None'")>]>
This commit explicitly sets replicaSet to None to indicate that it's a
standalone MongoDB instance. I also had to remove the CONTENTSTORE entry
from auth.yml because edx-platform's devstack.py assumes it has a
non-null value (set in common.py), and devstack.py executes before
tutor's development.py can set this replicaSet value.
- [Feature] Add the `-m/--mount` option to `tutor dev quickstart`.
- [Bugfix] Fix `tutor dev start -m /path/to/frontend-app-learning` by introducing dev-specific `COMPOSE_DEV_TMP` and `COMPOSE_DEV_JOBS_TMP` filters (by @regisb).
- [Bugfix] Log the shell commands that Tutor executes more accurately. (by @kdmccormick)
- [Bugfix] `tutor dev quickstart` would fail under certain versions of docker-compose due to a bug in the logic that handled volume mounting. (by @kdmccormick)
- [Bugfix] The `tutor k8s start` command will succeed even when `k8s-override` and `kustomization-patches-strategic-merge` are not specified. (by @edazzocaisser)
- [BugFix] `kubectl wait` checks deployments instead of pods as it could hang indefinitely if there are extra pods in a broken state. (by @keithgg)
The -m/--mount option makes it possible to bind-mount volumes at runtime. The
volumes are declared in a local/docker-compose.tmp.yml file. The problem with
this approach is when we want to bind-mount a volume to a service which is
specific to the dev context. For instance: the "learning" service when the MFE
plugin is enabled.
In such a case, starting the service triggers a call to `docker-compose stop`
in the local context. This call fails because the "learning" service does not
exist in the local context. Note that this issue only seems to occur with
docker-compose v1.
To resolve this issue, we create two additional filters for
the dev context, which emulate the behaviour of the local context. With this approach, we convert the -m/--mount arguments right after they are parsed. Because they are parsed just once, we can get rid of the de-duplication logic initially introduced with the COMPOSE_CLI_MOUNTS context.
Close #711. Close also https://github.com/overhangio/tutor-mfe/issues/57.
Whenever Tutor executes a shell command, it logs out said
command in order to aid in end user understanding/debugging.
In some cases (notably, when running jobs in containers)
the logged command was not accurately quoted. The command
was run correctly, because it was passed in pieces to
``subprocess.Popen``, which correctly joins the pieces together
into a valid POSIX shell command; however, the logged version
of the command was constructed by simply joining the pieces
with spaces. This usually works, but breaks down when running
complex shell commands with nested quoting.
This commit changes the logging to use ``shlex.join``, which
joins command pieces together in a POSIX-compliant way,
presumably the same way as ``subprocess.Popen``.
Example:
tutor local importdemocourse
runs the shell command:
docker-compose -f /home/kyle/.local/share/tutor/env/local/docker-compose.yml -f /home/kyle/.local/share/tutor/env/local/docker-compose.prod.yml -f /home/kyle/.local/share/tutor/env/local/docker-compose.tmp.yml --project-name tutor_local -f /home/kyle/.local/share/tutor/env/local/docker-compose.jobs.yml -f /home/kyle/.local/share/tutor/env/local/docker-compose.jobs.tmp.yml run --rm cms-job sh -e -c 'echo "Loading settings $DJANGO_SE... (several more script lines) ...eindex_course --all --setup'
but the logged shell command was:
docker-compose -f /home/kyle/.local/share/tutor/env/local/docker-compose.yml -f /home/kyle/.local/share/tutor/env/local/docker-compose.prod.yml -f /home/kyle/.local/share/tutor/env/local/docker-compose.tmp.yml --project-name tutor_local -f /home/kyle/.local/share/tutor/env/local/docker-compose.jobs.yml -f /home/kyle/.local/share/tutor/env/local/docker-compose.jobs.tmp.yml run --rm cms-job sh -e -c echo "Loading settings $DJANGO_SE... (several more script lines) ...eindex_course --all --setup
which will not run if copied and pasted back into the
user's terminal, as the importdemocourse shell script is unquoted.
When waiting for pods, it's possible that the deployment may be
complete but, because other pods may have been Evicted or Killed, the
wait wait condition completes.
In certain code paths, such as in `tutor local quickstart`,
`process_mount_points` is called more than once in the same process,
causing mounts to be added to `COMPOSE_LOCAL[_JOBS]_TMP` redundantly.
As a result, docker-compose[.jobs].tmp.yml was occasionally being
rendered with duplicate volume specifiers. Some versions of Docker
Compose ignored this; other versions warned or threw an error.
In order to make `process_mount_points` tolerant to being called
multiple times, we wrap its volume-adding callbacks within a new
hooks context. This allows us to clear said hooks context every
time `process_mount_points` is called, essentially making the
function idempotent.
Co-authored-by: Régis Behmo <regis@behmo.com>
- [Bugfix] Build openedx-dev Docker image even when the host user is root, for instance on Windows. (by @regisb)
- [Bugfix] Patch nutmeg.1 release with [LTI 1.3 fix](https://github.com/openedx/edx-platform/pull/30716). (by @ormsbee)
- [Improvement] Make it possible to override k8s resources in plugins using `k8s-override` patch. (by @foadlind)
Sometimes, the host user is root: this may happen when tutor is run with
"sudo" (which is not recommended) or on Windows. In such cases, building
the image should not fail, but default to a reasonable user. Also, when
we pass an invalid APP_USER_ID as a build arg, then we should fail with
an explicit message.
See this conversation:
https://discuss.overhang.io/t/problem-with-dev-image-build-useradd-uid-0-is-not-unique/2406
The pymongo dependency for edx-platform was updated (3.10.1 to 3.12.3)
in https://github.com/openedx/edx-platform/pull/30569
This caused the following error when running the edx-platform database
migration split_modulestore_django.0002_data_migration as part of
`tutor dev quickstart`:
pymongo.errors.ServerSelectionTimeoutError: client is configured to
connect to a replica set named '' but this node belongs to a set named
'None', Timeout: 30s, Topology Description: <TopologyDescription id:
62bdbaf182687350acf1aeec, topology_type: Single, servers:
[<ServerDescription ('mongodb', 27017) server_type: Unknown, rtt:
None, error=ConfigurationError("client is configured to connect to a
replica set named '' but this node belongs to a set named 'None'")>]>
This commit explicitly sets replicaSet to None to indicate that it's a
standalone MongoDB instance. I also had to remove the CONTENTSTORE entry
from auth.yml because edx-platform's devstack.py assumes it has a
non-null value (set in common.py), and devstack.py executes before
tutor's development.py can set this replicaSet value.
Currently there is no way for plugins to customize Kubernetes resources
defined in Tutor deployment manifests.
This change makes that possible by taking advantage of the strategic
merge patching mechanism in `kustomization.yml`.
Any resource definition in a `k8s-override` patch in a plugin will
override the resource defined by Tutor, provided that their names match.
Reference: https://github.com/overhangio/tutor/pull/675
- [Bugfix] Update problem with hint template so it works with newer python versions. (by @mariajgrimaldi)
- [Feature] Add default PYTHONBREAKPOINT to openedx/Dockerfile (by @Carlos-Muniz)
- [Bugfix] Fix smtp server port in `cms.yml` which was causing email sending failures in the Studio. (by @regisb)
- [Bugfix] Skip waiting for MongoDB if it is served using SRV records. (by @gabor-boros)
- [Improvement] Use `git am` instead of `cherry-pick` to simplify patching process.
- [Improvement] Tutor is now compatible with Docker Compose subcommand.
