Merge branch 'master' into nightly

CHANGELOG.md

@@ -17,6 +17,14 @@ Every user-facing change should have an entry in this changelog. Please respect
 -->
 
 ## Unreleased
+
+- [Fix] `tutor dev quickstart` would fail under certain versions of docker-compose due to a bug in the logic that handled volume mounting. (by @kdmccormick)
+- [Bugfix] The `tutor k8s start` command will succeed even when `k8s-override` and `kustomization-patches-strategic-merge` are not specified. (by @edazzocaisser)
+
+## v14.0.3 (2022-07-09)
+
+- [Bugfix] Build openedx-dev Docker image even when the host user is root, for instance on Windows. (by @regisb)
+- [Bugfix] Patch nutmeg.1 release with [LTI 1.3 fix](https://github.com/openedx/edx-platform/pull/30716). (by @ormsbee)
 - [Improvement] Make it possible to override k8s resources in plugins using `k8s-override` patch. (by @foadlind)
 
 ## v14.0.2 (2022-06-27)

docs/configuration.rst

@@ -63,6 +63,38 @@ Custom images
 
 This configuration parameter defines the name of the Docker image to run for the lms and cms containers. By default, the Docker image tag matches the Tutor version it was built with.
 
+- ``DOCKER_IMAGE_OPENEDX_DEV`` (default: ``"openedx-dev"``)
+
+This configuration paramater defines the name of the Docker image to run the development version of the lms and cms containers.  By default, the Docker image tag matches the Tutor version it was built with.
+
+- ``DOCKER_IMAGE_CADDY`` (default: ``"docker.io/caddy:2.4.6"``)
+
+This configuration paramater defines which Caddy Docker image to use.
+
+- ``DOCKER_IMAGE_ELASTICSEARCH`` (default: ``"docker.io/elasticsearch:7.10.1"``)
+
+This configuration parameter defines which Elasticsearch Docker image to use.
+
+- ``DOCKER_IMAGE_MONGODB`` (default: ``"docker.io/mongo:4.2.17"``)
+
+This configuration parameter defines which MongoDB Docker image to use.
+
+- ``DOCKER_IMAGE_MYSQL`` (default: ``"docker.io/mysql:5.7.35"``)
+
+This configuration parameter defines which MySQL Docker image to use.
+
+- ``DOCKER_IMAGE_REDIS`` (default: ``"docker.io/redis:6.2.6"``)
+
+This configuration parameter defines which Redis Docker image to use.
+
+- ``DOCKER_IMAGE_SMTP`` (default: ``"docker.io/devture/exim-relay:4.95-r0-2``)
+
+This configuration parameter defines which Simple Mail Transfer Protocol (SMTP) Docker image to use.
+
+- ``DOCKER_IMAGE_PERMISSIONS`` (default: ``"{{ DOCKER_REGISTRY }}overhangio/openedx-permissions:{{ TUTOR_VERSION }}"``)
+
+This configuration parameter defines the Docker image to be used for setting file permissions. The default image sets all containers to be run as unpriveleged users.
+
 Custom registry
 ***************
 
@@ -76,6 +108,21 @@ You may want to pull/push images from/to a custom docker registry. For instance,
 
 .. _openedx_configuration:
 
+Compose
+*******
+
+- ``DOCKER_COMPOSE_VERSION`` (default: ``"3.7"``)
+
+This configuration parameter sets the version of Docker Compose to be used to build all containers.
+
+- ``DEV_PROJECT_NAME`` (default: ``"{{ TUTOR_APP }}_dev"``)
+
+This configuration parameter sets the Development version of the Docker Compose project name.
+
+- ``LOCAL_PROJECT_NAME`` (default: ``"{{ TUTOR_APP }}_local"``)
+
+This configuration parameter sets the Local version of the Docker Compose project name.
+
 Open edX customisation
 ~~~~~~~~~~~~~~~~~~~~~~
 
@@ -111,6 +158,47 @@ This defines extra pip packages that are going to be installed for Open edX.
 
 This defines the registry from which you'll be pulling NPM packages when building Docker images. Like ``EDX_PLATFORM_REPOSITORY``, this can be overridden at build time with a ``--build-arg`` option.
 
+- ``OPENEDX_AWS_ACCESS_KEY`` (default: ``""``)
+
+This configuration parameter sets the Django setting ``AWS_ACCESS_KEY_ID`` in edx-platform's LMS, CMS, envs, and production.py for use by the library django-storages with Amazon S3.
+
+- ``OPENEDX_AWS_SECRET_ACCESS_KEY`` (default: ``""``)
+
+This configuration parameter sets the Django setting ``AWS_SECRET_ACCESS_KEY`` in edx-platform's LMS, CMS, envs, and production.py for use by the library django-storages with Amazon S3.
+
+- ``OPENEDX_MYSQL_DATABASE`` (default: ``"openedx"``)
+
+This configuration parameter sets the name of the MySQL Database to be used by the Open edX Instance.
+
+- ``OPENEDX_CSMH_MYSQL_DATABASE`` (default: ``"{{ OPENEDX_MYSQL_DATABASE }}_csmh"``)
+
+This configuration parameter allows you to configure the name of the separate Courseware Student-Module History (CSMH) database.
+
+- ``OPENEDX_MYSQL_USERNAME`` (default: ``"openedx"``)
+
+This configuration parameter sets the username associated with the MySQL Database.
+
+CMS OAUTH2 SSO
+~~~~~~~~~~~~~~
+
+- ``CMS_OAUTH2_KEY_SSO`` (default: ``"cms-sso"``)
+
+This defines the Studio's (CMS) OAUTH 2.0 Login (Key or Client ID) for SSO in the production environment.
+
+- ``CMS_OAUTH2_KEY_SSO_DEV`` (default: ``"cms-sso-dev"``)
+
+This defines the Studio's (CMS) OAUTH 2.0 Login (Key or Client ID) for SSO in the development environment.
+
+For more information, see `Enabling OAuth for Studio login <https://github.com/openedx/edx-platform/blob/master/docs/guides/studio_oauth.rst>`__.
+
+JWTs
+~~~~
+
+- ``JWT_COMMON_AUDIENCE`` (default: ``"openedx"``)
+- ``JWT_COMMON_ISSUER`` (default: ``"{% if ENABLE_HTTPS %}https{% else %}http{% endif %}://{{ LMS_HOST }}/oauth2"``)
+- ``JWT_COMMON_SECRET_KEY`` (default: ``"{{ OPENEDX_SECRET_KEY }}"``)
+
+These configuration parameters are rendered into the ``JWT_AUTH`` dictionary with keys ``JWT_AUDIENCE``, ``JWT_ISSUER``, and ``JWT_SECRET_KEY``, respectively. These parameters may be changed in order to create a custom user login for testing purposes.
 
 Vendor services
 ~~~~~~~~~~~~~~~
@@ -205,6 +293,24 @@ If you would like to perform SSL/TLS termination with your own custom certificat
 
 .. _custom_openedx_docker_image:
 
+Kubernetes
+~~~~~~~~~~
+
+- ``K8S_NAMESPACE`` (default: ``"openedx"``)
+
+This configuration parameter sets the Kubernetes Namespace.
+
+Miscellaneous Project Settings
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+- ``CONTACT_EMAIL`` (default: ``"contact@{{ LMS_HOST }}"``)
+
+This configuration parameter sets the Contact Email.
+
+- ``PLATFORM_NAME`` (default: ``"My Open edX"``)
+
+This configuration parameter sets the Platform Name.
+
 Custom Open edX docker image
 ----------------------------
 
@@ -303,7 +409,7 @@ If you don't create your fork from this tag, you *will* have important compatibi
 Adding custom translations
 ~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-If you are not running Open edX in English, chances are that some strings will not be properly translated. In most cases, this is because not enough contributors have helped translate Open edX into your language. It happens! With Tutor, available translated languages include those that come bundled with `edx-platform <https://github.com/openedx/edx-platform/tree/open-release/nutmeg.master/conf/locale>`__ as well as those from `openedx-i18n <https://github.com/openedx/openedx-i18n/tree/master/edx-platform/locale>`__.
+If you are not running Open edX in English (``LANGUAGE_CODE`` default: ``"en"``), chances are that some strings will not be properly translated. In most cases, this is because not enough contributors have helped translate Open edX into your language. It happens! With Tutor, available translated languages include those that come bundled with `edx-platform <https://github.com/openedx/edx-platform/tree/open-release/nutmeg.master/conf/locale>`__ as well as those from `openedx-i18n <https://github.com/openedx/openedx-i18n/tree/master/edx-platform/locale>`__.
 
 Tutor offers a relatively simple mechanism to add custom translations to the openedx Docker image. You should create a folder that corresponds to your language code in the "build/openedx/locale" folder of the Tutor environment. This folder should contain a "LC_MESSAGES" folder. For instance::
 

tests/commands/test_compose.py

@@ -1,11 +1,16 @@
+import typing as t
 import unittest
 
 from click.exceptions import ClickException
 
+from tutor import hooks
 from tutor.commands import compose
 
 
 class ComposeTests(unittest.TestCase):
+
+    maxDiff = None  # Ensure we can see long diffs of YAML files.
+
     def test_mount_option_parsing(self) -> None:
         param = compose.MountParam()
 
@@ -36,3 +41,49 @@ class ComposeTests(unittest.TestCase):
         )
         with self.assertRaises(ClickException):
             param("lms,:/path/to/edx-platform:/openedx/edx-platform")
+
+    def test_compose_local_tmp_generation(self) -> None:
+        """
+        Ensure that docker-compose.tmp.yml is correctly generated, even when
+        mounts are processed more than once.
+        """
+        param = compose.MountParam()
+        mount_args: t.Tuple[t.List[compose.MountParam.MountType], ...] = (
+            # Auto-mounting of edx-platform to lms* and cms*
+            param("/path/to/edx-platform"),
+            # Manual mounting of some other folder to mfe and lms
+            param("mfe,lms:/path/to/something-else:/openedx/something-else"),
+        )
+
+        # In some cases, process_mount_arguments ends up being called more
+        # than once. To ensure that we can handle that situation, we call it
+        # multiple times here.
+        compose.process_mount_arguments(mount_args)
+        compose.process_mount_arguments(mount_args)
+        compose.process_mount_arguments(mount_args)
+
+        compose_file: t.Dict[str, t.Any] = hooks.Filters.COMPOSE_LOCAL_TMP.apply({})
+        actual_services: t.Dict[str, t.Any] = compose_file["services"]
+        expected_services: t.Dict[str, t.Any] = {
+            "cms": {"volumes": ["/path/to/edx-platform:/openedx/edx-platform"]},
+            "cms-worker": {"volumes": ["/path/to/edx-platform:/openedx/edx-platform"]},
+            "lms": {
+                "volumes": [
+                    "/path/to/edx-platform:/openedx/edx-platform",
+                    "/path/to/something-else:/openedx/something-else",
+                ]
+            },
+            "lms-worker": {"volumes": ["/path/to/edx-platform:/openedx/edx-platform"]},
+            "mfe": {"volumes": ["/path/to/something-else:/openedx/something-else"]},
+        }
+        self.assertEqual(actual_services, expected_services)
+
+        compose_jobs_file: t.Dict[
+            str, t.Any
+        ] = hooks.Filters.COMPOSE_LOCAL_JOBS_TMP.apply({})
+        actual_jobs_services: t.Dict[str, t.Any] = compose_jobs_file["services"]
+        expected_jobs_services: t.Dict[str, t.Any] = {
+            "cms-job": {"volumes": ["/path/to/edx-platform:/openedx/edx-platform"]},
+            "lms-job": {"volumes": ["/path/to/edx-platform:/openedx/edx-platform"]},
+        }
+        self.assertEqual(actual_jobs_services, expected_jobs_services)

tutor/__about__.py

@@ -2,7 +2,7 @@ import os
 
 # Increment this version number to trigger a new release. See
 # docs/tutor.html#versioning for information on the versioning scheme.
-__version__ = "14.0.2"
+__version__ = "14.0.3"
 
 # The version suffix will be appended to the actual version, separated by a
 # dash. Use this suffix to differentiate between the actual released version and

tutor/commands/compose.py

@@ -465,7 +465,7 @@ def dc_command(context: BaseComposeContext, command: str, args: t.List[str]) ->
     context.job_runner(config).docker_compose(command, *volume_args, *non_volume_args)
 
 
-def process_mount_arguments(mounts: t.Tuple[t.List[MountParam.MountType]]) -> None:
+def process_mount_arguments(mounts: t.Tuple[t.List[MountParam.MountType], ...]) -> None:
     """
     Process --mount arguments.
 
@@ -496,18 +496,31 @@ def process_mount_arguments(mounts: t.Tuple[t.List[MountParam.MountType]]) -> No
             services[service]["volumes"].append(f"{host_path}:{container_path}")
         return docker_compose
 
-    # Save bind-mounts
-    @hooks.Filters.COMPOSE_LOCAL_TMP.add()
-    def _add_mounts_to_docker_compose_tmp(
-        docker_compose_tmp: t.Dict[str, t.Any]
-    ) -> t.Dict[str, t.Any]:
-        return _add_mounts(docker_compose_tmp, app_mounts)
+    # Clear filter callbacks already created within the COMPOSE_CLI_MOUNTS context.
+    # This prevents us from redundantly specifying these volume mounts in cases
+    # where process_mount_arguments is called more than once.
+    hooks.Filters.COMPOSE_LOCAL_TMP.clear(
+        context=hooks.Contexts.COMPOSE_CLI_MOUNTS.name
+    )
+    hooks.Filters.COMPOSE_LOCAL_JOBS_TMP.clear(
+        context=hooks.Contexts.COMPOSE_CLI_MOUNTS.name
+    )
 
-    @hooks.Filters.COMPOSE_LOCAL_JOBS_TMP.add()
-    def _add_mounts_to_docker_compose_jobs_tmp(
-        docker_compose_tmp: t.Dict[str, t.Any]
-    ) -> t.Dict[str, t.Any]:
-        return _add_mounts(docker_compose_tmp, job_mounts)
+    # Now, within that COMPOSE_CLI_MOUNTS context, (re-)create filter callbacks to
+    # specify these volume mounts within the docker-compose[.jobs].tmp.yml files.
+    with hooks.Contexts.COMPOSE_CLI_MOUNTS.enter():
+
+        @hooks.Filters.COMPOSE_LOCAL_TMP.add()
+        def _add_mounts_to_docker_compose_tmp(
+            docker_compose_tmp: t.Dict[str, t.Any]
+        ) -> t.Dict[str, t.Any]:
+            return _add_mounts(docker_compose_tmp, app_mounts)
+
+        @hooks.Filters.COMPOSE_LOCAL_JOBS_TMP.add()
+        def _add_mounts_to_docker_compose_jobs_tmp(
+            docker_compose_jobs_tmp: t.Dict[str, t.Any]
+        ) -> t.Dict[str, t.Any]:
+            return _add_mounts(docker_compose_jobs_tmp, job_mounts)
 
 
 @hooks.Filters.COMPOSE_MOUNTS.add()

tutor/hooks/consts.py

@@ -354,8 +354,12 @@ class Contexts:
     #: Plugins will be installed and enabled within this context.
     PLUGINS = contexts.Context("plugins")
 
-    #: YAML-formatted v0 plugins will be installed within that context.
+    #: YAML-formatted v0 plugins will be installed within this context.
     PLUGINS_V0_YAML = contexts.Context("plugins:v0:yaml")
 
-    #: Python entrypoint plugins will be installed within that context.
+    #: Python entrypoint plugins will be installed within this context.
     PLUGINS_V0_ENTRYPOINT = contexts.Context("plugins:v0:entrypoint")
+
+    #: Docker Compose volumes added via the CLI's ``--mount`` option will
+    #: be installed within this context.
+    COMPOSE_CLI_MOUNTS = contexts.Context("compose:cli:mounts")

tutor/templates/build/openedx/Dockerfile

@@ -131,7 +131,9 @@ RUN apt update && \
     rm -rf /var/lib/apt/lists/*
 
 # From then on, run as unprivileged "app" user
+# Note that this must always be different from root (APP_USER_ID=0)
 ARG APP_USER_ID=1000
+RUN if [ "$APP_USER_ID" = 0 ]; then echo "app user may not be root" && false; fi
 RUN useradd --home-dir /openedx --create-home --shell /bin/bash --uid ${APP_USER_ID} app
 USER ${APP_USER_ID}
 

tutor/templates/dev/docker-compose.yml

@@ -7,7 +7,8 @@ x-openedx-service:
     context: ../build/openedx/
     target: development
     args:
-      APP_USER_ID: "{{ HOST_USER_ID }}"
+      # Note that we never build the openedx-dev image with root user ID, as it would simply fail.
+      APP_USER_ID: "{{ HOST_USER_ID or 1000 }}"
   stdin_open: true
   tty: true
   volumes:

tutor/templates/kustomization.yml

@@ -59,8 +59,12 @@ configMapGenerator:
         app.kubernetes.io/name: redis
 {{ patch("kustomization-configmapgenerator") }}
 
+{%- if patch("k8s-override") or patch("kustomization-patches-strategic-merge") %}
 patchesStrategicMerge:
+{%- if patch("k8s-override") %}
 - k8s/override.yml
+{%- endif %}
 {{ patch("kustomization-patches-strategic-merge") }}
+{%- endif %}
 
 {{ patch("kustomization") }}