Merge remote-tracking branch 'origin/master' into nightly

CHANGELOG.md

@@ -20,6 +20,15 @@ instructions, because git commits are used to generate release notes:
 
 <!-- scriv-insert-here -->
 
+<a id='changelog-16.1.8'></a>
+## v16.1.8 (2023-12-10)
+
+- [Feature] Make it easy to work on 3rd-party edx-platform Python packages with `tutor mounts add /path/to/my/package`. (by @regisb)
+- [Improvement] When configured with `RUN_MYSQL: true`, run `mysqld` with binlog expiry set to 3 days (rather than the default of 30).
+- [Improvement] Fix `ulimits` error for elasticsearch in Docker rootless mode (by @OmarIthawi)
+- [Improvement] Do not hardcode `OPENEDX_COMMON_VERSION = master` in the nightly branch. This removes git conflicts whenever we bump the common version in the master branch. (by @regisb)
+- [Improvement] The ``iter_mounts`` template function can now take multiple image names as argument. This should concern only very advanced users. (by @regisb)
+
 <a id='changelog-16.1.7'></a>
 ## v16.1.7 (2023-11-17)
 

changelog.d/20231023_155558_fghaas_binlog_expiry.md

@@ -1 +0,0 @@
-[Improvement] When configured with `RUN_MYSQL: true`, run `mysqld` with binlog expiry set to 3 days (rather than the default of 30).

changelog.d/20231116_081945_i_rootless_docker.md

@@ -1,2 +0,0 @@
-- [Improvement] Fix `ulimits` error for elasticsearch in Docker rootless mode (by @OmarIthawi)
-

changelog.d/20231120_160115_regis_nightly_common_version.md

@@ -1 +0,0 @@
-- [Improvement] Do not hardcode `OPENEDX_COMMON_VERSION = master` in the nightly branch. This removes git conflicts whenever we bump the common version in the master branch. (by @regisb)

docs/dev.rst

@@ -5,6 +5,8 @@ Open edX development
 
 In addition to running Open edX in production, Tutor can be used for local development of Open edX. This means that it is possible to hack on Open edX without setting up a Virtual Machine. Essentially, this replaces the devstack provided by edX.
 
+For detailed explanations on how to work on edx-platform and its dependencies, see the :ref:`edx_platform` tutorial.
+
 .. _edx_platform_dev_env:
 
 First-time setup
@@ -12,7 +14,7 @@ First-time setup
 
 Firstly, either :ref:`install Tutor <install>` (for development against the named releases of Open edX) or :ref:`install Tutor Nightly <nightly>` (for development against Open edX's master branches).
 
-Then, optionally, tell Tutor to use a local fork of edx-platform.::
+Then, optionally, tell Tutor to use a local fork of edx-platform::
 
     tutor mounts add ./edx-platform
 
@@ -28,7 +30,6 @@ This will perform several tasks. It will:
 * disable HTTPS,
 * set ``LMS_HOST`` to `local.edly.io <http://local.edly.io>`_ (a convenience domain that simply `points at 127.0.0.1 <https://dnschecker.org/#A/local.edly.io>`_),
 * prompt for a platform details (with suitable defaults),
-* build an ``openedx-dev`` image,
 * start LMS, CMS, supporting services, and any plugged-in services,
 * ensure databases are created and migrated, and
 * run service initialization scripts, such as service user creation and Waffle configuration.
@@ -83,20 +84,6 @@ Finally, the platform can also be started back up with ``launch``. It will take
 
   tutor dev launch --pullimages
 
-Debugging with breakpoints
---------------------------
-
-To debug a local edx-platform repository, first, start development in detached mode (with ``-d``), add a `python breakpoint <https://docs.python.org/3/library/functions.html#breakpoint>`__ with ``breakpoint()`` anywhere in the code. Then, attach to the applicable service's container by running ``start`` (without ``-d``) followed by the service's name::
-
-  # Start in detached mode:
-  tutor dev start -d
-
-  # Debugging LMS:
-  tutor dev start lms
-
-  # Or, debugging CMS:
-  tutor dev start cms
-
 Running arbitrary commands
 --------------------------
 
@@ -234,7 +221,7 @@ You can then edit the files in ``~/venv`` on your local filesystem and see the c
 Manual bind-mount to any directory
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-.. warning:: Manually bind-mounting volumes with the ``--volume`` option makes it difficult to simultaneously bind-mount to multiple containers. Also, the ``--volume`` options are not compatible with ``start`` commands. For an alternative, see the :ref:`persistent mounts <persistent_mounts>`.
+.. warning:: Manually bind-mounting volumes with the ``--volume`` option makes it difficult to simultaneously bind-mount to multiple containers. Also, the ``--volume`` options are not compatible with ``start`` commands. As an alternative, you should consider following the instructions above: :ref:`persistent_mounts`.
 
 The above solution may not work for you if you already have an existing directory, outside of the "volumes/" directory, which you would like mounted in one of your containers. For instance, you may want to mount your copy of the `edx-platform <https://github.com/openedx/edx-platform/>`__ repository. In such cases, you can simply use the ``-v/--volume`` `Docker option <https://docs.docker.com/storage/volumes/#choose-the--v-or---mount-flag>`__::
 
@@ -243,6 +230,8 @@ The above solution may not work for you if you already have an existing director
 Override docker-compose volumes
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
+.. warning:: While the option described below "works", it will only bind-mount directories at run-time. In many cases you really want to bind-mount directories at build-time. For instance: when working on edx-platform requirements. As an alternative, you should consider following the instructions above: :ref:`persistent_mounts`.
+
 Adding items to the ``MOUNTS`` setting effectively adds new bind-mount volumes to the ``docker-compose.yml`` files. But you might want to have more control over your volumes, such as adding read-only options, or customising other fields of the different services. To address these issues, you can create a ``docker-compose.override.yml`` file that will specify custom volumes to be used with all ``dev`` commands::
 
     vim "$(tutor config printroot)/env/dev/docker-compose.override.yml"
@@ -268,55 +257,3 @@ This override file will be loaded when running any ``tutor dev ..`` command. The
 
 .. note::
     The ``tutor local`` commands load the ``docker-compose.override.yml`` file from the ``$(tutor config printroot)/env/local/docker-compose.override.yml`` directory. One-time jobs from initialisation commands load the ``local/docker-compose.jobs.override.yml`` and ``dev/docker-compose.jobs.override.yml``.
-
-Common tasks
-------------
-
-XBlock and edx-platform plugin development
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-In some cases, you will have to develop features for packages that are pip-installed next to the edx-platform. This is quite easy with Tutor. Just add your packages to the ``$(tutor config printroot)/env/build/openedx/requirements/private.txt`` file. To avoid re-building the openedx Docker image at every change, you should add your package in editable mode. For instance::
-
-    echo "-e ./mypackage" >> "$(tutor config printroot)/env/build/openedx/requirements/private.txt"
-
-The ``requirements`` folder should have the following content::
-
-    env/build/openedx/requirements/
-        private.txt
-        mypackage/
-            setup.py
-            ...
-
-You will have to re-build the openedx Docker image once::
-
-    tutor images build openedx
-
-You should then run the development server as usual, with ``start``. Every change made to the ``mypackage`` folder will be picked up and the development server will be automatically reloaded.
-
-Running edx-platform unit tests
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-It's possible to run the full set of unit tests that ship with `edx-platform <https://github.com/openedx/edx-platform/>`__. To do so, run a shell in the LMS development container::
-
-    tutor dev run lms bash
-
-Then, run unit tests with ``pytest`` commands::
-
-    # Run tests on common apps
-    unset DJANGO_SETTINGS_MODULE
-    unset SERVICE_VARIANT
-    export EDXAPP_TEST_MONGO_HOST=mongodb
-    pytest common
-    pytest openedx
-    pytest xmodule
-
-    # Run tests on LMS
-    export DJANGO_SETTINGS_MODULE=lms.envs.tutor.test
-    pytest lms
-
-    # Run tests on CMS
-    export DJANGO_SETTINGS_MODULE=cms.envs.tutor.test
-    pytest cms
-
-.. note::
-    Getting all edx-platform unit tests to pass on Tutor is currently a work-in-progress. Some unit tests are still failing. If you manage to fix some of these, please report your findings in the `Open edX forum <https://discuss.openedx.org/tag/tutor>`__.

docs/reference/api/hooks/catalog.rst

@@ -16,3 +16,9 @@ The underlying Python hook classes and API are documented :ref:`here <hooks_api>
 
 .. autoclass:: tutor.hooks.Contexts
     :members:
+
+Open edX hooks
+--------------
+
+.. automodule:: tutor.plugins.openedx.hooks
+    :members:

docs/tutorials/edx-platform.rst

@@ -0,0 +1,177 @@
+.. _edx_platform:
+
+Working on edx-platform as a developer
+======================================
+
+Tutor supports running in development with ``tutor dev`` commands. Developers frequently need to work on a fork of some repository. The question then becomes: how to make their changes available within the "openedx" Docker container? 
+
+For instance, when troubleshooting an issue in `edx-platform <https://github.com/openedx/edx-platform>`__, we would like to make some changes to a local fork of that repository, and then apply these changes immediately in the "lms" and the "cms" containers (but also "lms-worker", "cms-worker", etc.)
+
+Similarly, when developing a custom XBlock, we would like to hot-reload any change we make to the XBlock source code within the containers.
+
+Tutor provides a simple solution to these questions. In both cases, the solution takes the form of a ``tutor mounts add ...`` command.
+
+Working on the "edx-platform" repository
+----------------------------------------
+
+Download the code from the upstream repository::
+
+    cd /my/workspace/edx-plaform
+    git clone https://github.com/openedx/edx-platform .
+
+Check out the right version of the upstream repository. If you are working on the `current "zebulon" release <https://docs.openedx.org/en/latest/community/release_notes/index.html>`__ of Open edX, then you should checkout the corresponding branch::
+
+    # "zebulon" is an example. You should put the actual release name here.
+    # I.e: aspen, birch, cypress, etc.
+    git checkout open-release/zebulon.master
+
+On the other hand, if you are working on the Tutor :ref:`"nightly" <nightly>` branch then you should checkout the master branch::
+
+    git checkout master
+
+Then, mount the edx-platform repository with Tutor::
+
+    tutor mounts add /my/workspace/edx-plaform
+
+This command does a few "magical" things 🧙 behind the scenes:
+
+1. Mount the edx-platform repository in the image at build-time. This means that when you run ``tutor images build openedx``, your custom repository will be used instead of the upstream. In particular, any change you've made to the installed requirements, static assets, etc. will be taken into account.
+2. Mount the edx-platform repository at run time. Thus, when you run ``tutor dev start``, any change you make to the edx-platform repository will be hot-reloaded.
+
+You can get a glimpse of how these auto-mounts work by running ``tutor mounts list``. It should output something similar to the following::
+
+    $ tutor mounts list
+    - name: /home/data/regis/projets/overhang/repos/edx/edx-platform
+    build_mounts:
+    - image: openedx
+        context: edx-platform
+    - image: openedx-dev
+        context: edx-platform
+    compose_mounts:
+    - service: lms
+        container_path: /openedx/edx-platform
+    - service: cms
+        container_path: /openedx/edx-platform
+    - service: lms-worker
+        container_path: /openedx/edx-platform
+    - service: cms-worker
+        container_path: /openedx/edx-platform
+    - service: lms-job
+        container_path: /openedx/edx-platform
+    - service: cms-job
+        container_path: /openedx/edx-platform
+
+Working on edx-platform Python dependencies
+-------------------------------------------
+
+Quite often, developers don't want to work on edx-platform directly, but on a dependency of edx-platform. For instance: an XBlock. This works the same way as above. Let's take the example of the `"edx-ora2" <https://github.com/openedx/edx-ora2>`__ package, for open response assessments. First, clone the Python package::
+
+    cd /my/workspace/edx-ora2
+    git clone https://github.com/openedx/edx-ora2 .
+
+Then, check out the right version of the package. This is the version that is indicated in the `edx-platform/requirements/edx/base.txt <https://github.com/openedx/edx-platform/blob/open-release/palm.master/requirements/edx/base.txt>`__. Be careful that the version that is currently in use in your version of edx-platform is **not necessarily the head of the master branch**::
+
+    git checkout <my-version-tag-or-branch>
+
+Then, mount this repository::
+
+    tutor mounts add /my/workspace/edx-ora2
+
+Verify that your repository is properly bind-mounted by running ``tutor mounts list``::
+
+    $ tutor mounts list
+    - name: /my/workspace/edx-ora2
+    build_mounts:
+    - image: openedx
+        context: mnt-edx-ora2
+    - image: openedx-dev
+        context: mnt-edx-ora2
+    compose_mounts:
+    - service: lms
+        container_path: /mnt/edx-ora2
+    - service: cms
+        container_path: /mnt/edx-ora2
+    - service: lms-worker
+        container_path: /mnt/edx-ora2
+    - service: cms-worker
+        container_path: /mnt/edx-ora2
+    - service: lms-job
+        container_path: /mnt/edx-ora2
+    - service: cms-job
+        container_path: /mnt/edx-ora2
+
+You should then re-build the "openedx" Docker image to pick up your changes::
+
+    tutor images build openedx-dev
+
+Then, whenever you run ``tutor dev start``, the "lms" and "cms" container should automatically hot-reload your changes.
+
+To push your changes in production, you should do the same with ``tutor local`` and the "openedx" image::
+
+    tutor images build openedx
+    tutor local start -d
+
+Debugging with breakpoints
+--------------------------
+
+To debug a local edx-platform repository, first, start development in detached mode (with ``-d``), add a `python breakpoint <https://docs.python.org/3/library/functions.html#breakpoint>`__ with ``breakpoint()`` anywhere in the code. Then, attach to the applicable service's container by running ``start`` (without ``-d``) followed by the service's name::
+
+  # Start in detached mode:
+  tutor dev start -d
+
+  # Debugging LMS:
+  tutor dev start lms
+
+  # Or, debugging CMS:
+  tutor dev start cms
+
+Running edx-platform unit tests
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+It's possible to run the full set of unit tests that ship with `edx-platform <https://github.com/openedx/edx-platform/>`__. To do so, run a shell in the LMS development container::
+
+    tutor dev run lms bash
+
+Then, run unit tests with ``pytest`` commands::
+
+    # Run tests on common apps
+    unset DJANGO_SETTINGS_MODULE
+    unset SERVICE_VARIANT
+    export EDXAPP_TEST_MONGO_HOST=mongodb
+    pytest common
+    pytest openedx
+    pytest xmodule
+
+    # Run tests on LMS
+    export DJANGO_SETTINGS_MODULE=lms.envs.tutor.test
+    pytest lms
+
+    # Run tests on CMS
+    export DJANGO_SETTINGS_MODULE=cms.envs.tutor.test
+    pytest cms
+
+.. note::
+    Getting all edx-platform unit tests to pass on Tutor is currently a work-in-progress. Some unit tests are still failing. If you manage to fix some of these, please report your findings in the `Open edX forum <https://discuss.openedx.org/tag/tutor>`__.
+
+What if my edx-platform package is not automatically bind-mounted?
+------------------------------------------------------------------
+
+It is quite possible that your package is not automatically recognized and bind-mounted by Tutor. Out of the box, Tutor defines a set of regular expressions: if your package name matches this regular expression, it will be automatically bind-mounted. But if it does not, you have to tell Tutor about it.
+
+To do so, you will need to create a :ref:`Tutor plugin <plugin_development_tutorial>` that implements the :py:data:`tutor.hooks.Filters.MOUNTED_DIRECTORIES` filter::
+
+    import tutor import hooks
+    hooks.Filters.MOUNTED_DIRECTORIES.add_item(("openedx", "my-package"))
+
+After you implement and enable that plugin, ``tutor mounts list`` should display your directory among the bind-mounted directories.
+
+Do I have to re-build the "openedx" Docker image after every change?
+--------------------------------------------------------------------
+
+No, you don't. Re-building the "openedx" Docker image may take a while, and you don't want to run this command every time you make a change to your local repositories. Because your host directory is bind-mounted in the containers at runtime, your changes will be automatically applied to the container. If you run ``tutor dev`` commands, then your changes will be automatically picked up.
+
+If you run ``tutor local`` commands (for instance: when debugging a production instance) then your changes will *not* be automatically picked up. In such a case you should manually restart the containers::
+
+    tutor local restart lms cms lms-worker cms-worker
+
+Re-building the "openedx" image should only be necessary when you want to push your changes to a Docker registry, then pull them on a remote server.

docs/tutorials/index.rst

@@ -9,6 +9,7 @@ Open edX customization
 
    plugin
    theming
+   edx-platform
    edx-platform-settings
    google-smtp
    nightly

tests/test_bindmount.py

@@ -53,13 +53,6 @@ class BindmountTests(unittest.TestCase):
         import tutor.commands.compose
 
         self.assertEqual(
-            [
+            [("openedx", "/path/to/edx-platform", "/openedx/edx-platform")],
-                ("lms", "/path/to/edx-platform", "/openedx/edx-platform"),
-                ("cms", "/path/to/edx-platform", "/openedx/edx-platform"),
-                ("lms-worker", "/path/to/edx-platform", "/openedx/edx-platform"),
-                ("cms-worker", "/path/to/edx-platform", "/openedx/edx-platform"),
-                ("lms-job", "/path/to/edx-platform", "/openedx/edx-platform"),
-                ("cms-job", "/path/to/edx-platform", "/openedx/edx-platform"),
-            ],
             bindmount.parse_implicit_mount("/path/to/edx-platform"),
         )

tutor/__about__.py

@@ -2,7 +2,7 @@ import os
 
 # Increment this version number to trigger a new release. See
 # docs/tutor.html#versioning for information on the versioning scheme.
-__version__ = "16.1.7"
+__version__ = "16.1.8"
 
 # The version suffix will be appended to the actual version, separated by a
 # dash. Use this suffix to differentiate between the actual released version and

tutor/bindmount.py

@@ -12,7 +12,7 @@ def get_mounts(config: types.Config) -> list[str]:
     return types.get_typed(config, "MOUNTS", list)
 
 
-def iter_mounts(user_mounts: list[str], name: str) -> t.Iterable[str]:
+def iter_mounts(user_mounts: list[str], *names: str) -> t.Iterable[str]:
     """
     Iterate on the bind-mounts that are available to any given compose service. The list
     of bind-mounts is parsed from `user_mounts` and we yield only those for service
@@ -23,7 +23,7 @@ def iter_mounts(user_mounts: list[str], name: str) -> t.Iterable[str]:
     """
     for user_mount in user_mounts:
         for service, host_path, container_path in parse_mount(user_mount):
-            if service == name:
+            if service in names:
                 yield f"{host_path}:{container_path}"
 
 

tutor/commands/compose.py

@@ -426,38 +426,6 @@ def dc_command(
     context.job_runner(config).docker_compose(command, *args)
 
 
-@hooks.Filters.COMPOSE_MOUNTS.add()
-def _mount_edx_platform(
-    volumes: list[tuple[str, str]], name: str
-) -> list[tuple[str, str]]:
-    """
-    When mounting edx-platform with `tutor mounts add /path/to/edx-platform`,
-    bind-mount the host repo in the lms/cms containers.
-    """
-    if name == "edx-platform":
-        path = "/openedx/edx-platform"
-        volumes += [
-            ("lms", path),
-            ("cms", path),
-            ("lms-worker", path),
-            ("cms-worker", path),
-            ("lms-job", path),
-            ("cms-job", path),
-        ]
-    return volumes
-
-
-@hooks.Filters.APP_PUBLIC_HOSTS.add()
-def _edx_platform_public_hosts(
-    hosts: list[str], context_name: t.Literal["local", "dev"]
-) -> list[str]:
-    if context_name == "dev":
-        hosts += ["{{ LMS_HOST }}:8000", "{{ CMS_HOST }}:8001"]
-    else:
-        hosts += ["{{ LMS_HOST }}", "{{ CMS_HOST }}"]
-    return hosts
-
-
 hooks.Filters.ENV_TEMPLATE_VARIABLES.add_item(("iter_mounts", bindmount.iter_mounts))
 
 

tutor/commands/images.py

@@ -266,20 +266,6 @@ def get_image_build_contexts(config: Config) -> dict[str, list[tuple[str, str]]]
     return build_contexts
 
 
-@hooks.Filters.IMAGES_BUILD_MOUNTS.add()
-def _mount_edx_platform(
-    volumes: list[tuple[str, str]], path: str
-) -> list[tuple[str, str]]:
-    """
-    Automatically add an edx-platform repo from the host to the build context whenever
-    it is added to the `MOUNTS` setting.
-    """
-    if os.path.basename(path) == "edx-platform":
-        volumes.append(("openedx", "edx-platform"))
-        volumes.append(("openedx-dev", "edx-platform"))
-    return volumes
-
-
 @click.command(short_help="Pull images from the Docker registry")
 @click.argument("image_names", metavar="image", type=PullImageNameParam(), nargs=-1)
 @click.pass_obj

tutor/hooks/catalog.py

@@ -337,6 +337,7 @@ class Filters:
     #: - ``is_buildkit_enabled``: a deprecated function which always returns ``True`` now. Will be removed after Quince.
     #: - ``iter_values_named``: a function to iterate on variables that start or end with a given string.
     #: - ``iter_mounts``: a function that yields compose-compatible bind-mounts for any given service.
+    #: - ``iter_mounted_directories``: iterate on bind-mounted directory names.
     #: - ``patch``: a function to incorporate extra content into a template.
     #:
     #: :parameter filters: list of (name, value) tuples.
@@ -397,6 +398,53 @@ class Filters:
     #: Parameters are the same as for :py:data:`IMAGES_PULL`.
     IMAGES_PUSH: Filter[list[tuple[str, str]], [Config]] = Filter()
 
+    #: List of directories that will be automatically bind-mounted in an image (at
+    #: build-time) and a container (at run-time).
+    #:
+    #: Whenever a user runs: ``tutor mounts add /path/to/name``, "name" will be matched to
+    #: the regular expressions in this filter. If it matches, then the directory will be
+    #: automatically bind-mounted in the matching Docker image at build time and run
+    #: time. At build-time, they will be added to a layer named "mnt-{name}". At
+    #: run-time, they wll be mounted in ``/mnt/<name>``.
+    #:
+    #: In the case of edx-platform, ``pip install -e .`` will be run in this directory
+    #: at build-time. And the same host directory will be bind-mounted in that location
+    #: at run time. This allows users to transparently work on edx-platform
+    #: dependencies, such as Python packages.
+    #:
+    #: By default, xblocks and some common edx-platform packages are already present in
+    #: this filter, and associated to the "openedx" image. Add your own Python
+    #: dependencies to this filter to make it easier for users to work on the
+    #: dependencies of your app.
+    #:
+    #: See the list of all edx-platform base requirements here:
+    #: https://github.com/openedx/edx-platform/blob/master/requirements/edx/base.txt
+    #:
+    #: This filter was mostly designed for edx-platform, but it can be used by any
+    #: Python-based Docker image as well. The Dockerfile must declare mounted layers::
+    #:
+    #:     {% for name in iter_mounted_directories(MOUNTS, "yourimage") %}
+    #:     FROM scratch as mnt-{{ name }}
+    #:     {% endfor %}
+    #:
+    #: Then, Python packages are installed with::
+    #:
+    #:     {% for name in iter_mounted_directories(MOUNTS, "yourimage") %}
+    #:     COPY --from=mnt-{{ name }} --chown=app:app / /mnt/{{ name }}
+    #:     RUN pip install -e "/mnt/{{ name }}"
+    #:     {% endfor %}
+    #:
+    #: And the docker-compose service must include the following::
+    #:
+    #:    volumes:
+    #:      {%- for mount in iter_mounts(MOUNTS, "yourimage") %}
+    #:      - {{ mount }}
+    #:      {%- endfor %}
+    #:
+    #: :parameter list[tuple[str, str]] name_regex: Each tuple is the name of an image and a
+    #:   regular expression. For instance: ``("openedx", r".*xblock.*")``.
+    MOUNTED_DIRECTORIES: Filter[list[tuple[str, str]], []] = Filter()
+
     #: List of plugin indexes that are loaded when we run ``tutor plugins update``. By
     #: default, the plugin indexes are stored in the user configuration. This filter makes
     #: it possible to extend and modify this list with plugins.
@@ -459,8 +507,8 @@ class Contexts:
     """
 
     #: Dictionary of name/contexts. Each value is a context that we enter whenever we
-    #: create hooks for a specific application or : : plugin. For instance, plugin
+    #: create hooks for a specific application or plugin. For instance, plugin
-    #: "myplugin" will be enabled within the "app:myplugin" : context.
+    #: "myplugin" will be enabled within the "app:myplugin" context.
     APP: dict[str, Context] = {}
 
     @classmethod

tutor/plugins/__init__.py

@@ -10,7 +10,7 @@ from tutor import exceptions, fmt, hooks
 from tutor.types import Config, get_typed
 
 # Import modules to trigger hook creation
-from . import v0, v1
+from . import openedx, v0, v1
 
 # Cache of plugin patches, for efficiency
 ENV_PATCHES_DICT: dict[str, list[str]] = {}

tutor/plugins/openedx.py

@@ -0,0 +1,134 @@
+from __future__ import annotations
+
+import os
+import re
+import typing as t
+
+from tutor import bindmount
+from tutor import hooks
+
+
+@hooks.Filters.APP_PUBLIC_HOSTS.add()
+def _edx_platform_public_hosts(
+    hosts: list[str], context_name: t.Literal["local", "dev"]
+) -> list[str]:
+    if context_name == "dev":
+        hosts += ["{{ LMS_HOST }}:8000", "{{ CMS_HOST }}:8001"]
+    else:
+        hosts += ["{{ LMS_HOST }}", "{{ CMS_HOST }}"]
+    return hosts
+
+
+@hooks.Filters.IMAGES_BUILD_MOUNTS.add()
+def _mount_edx_platform_build(
+    volumes: list[tuple[str, str]], path: str
+) -> list[tuple[str, str]]:
+    """
+    Automatically add an edx-platform repo from the host to the build context whenever
+    it is added to the `MOUNTS` setting.
+    """
+    if os.path.basename(path) == "edx-platform":
+        volumes += [
+            ("openedx", "edx-platform"),
+            ("openedx-dev", "edx-platform"),
+        ]
+    return volumes
+
+
+@hooks.Filters.COMPOSE_MOUNTS.add()
+def _mount_edx_platform_compose(
+    volumes: list[tuple[str, str]], name: str
+) -> list[tuple[str, str]]:
+    """
+    When mounting edx-platform with `tutor mounts add /path/to/edx-platform`,
+    bind-mount the host repo in the lms/cms containers.
+    """
+    if name == "edx-platform":
+        path = "/openedx/edx-platform"
+        volumes.append(("openedx", path))
+    return volumes
+
+
+# Auto-magically bind-mount xblock directories and some common dependencies.
+hooks.Filters.MOUNTED_DIRECTORIES.add_items(
+    [
+        ("openedx", r".*[xX][bB]lock.*"),
+        ("openedx", "edx-enterprise"),
+        ("openedx", "edx-ora2"),
+        ("openedx", "edx-search"),
+        ("openedx", r"platform-plugin-.*"),
+    ]
+)
+
+
+@hooks.Filters.MOUNTED_DIRECTORIES.add(priority=hooks.priorities.LOW)
+def _add_openedx_dev_mounted_python_packages(
+    image_regex: list[tuple[str, str]]
+) -> list[tuple[str, str]]:
+    """
+    Automatically add python packages to "openedx-dev" if they are already in the
+    "openedx" image.
+    """
+    for image, regex in image_regex:
+        if image == "openedx":
+            image_regex.append(("openedx-dev", regex))
+    return image_regex
+
+
+@hooks.Filters.IMAGES_BUILD_MOUNTS.add()
+def _mount_python_requirements_build(
+    volumes: list[tuple[str, str]], path: str
+) -> list[tuple[str, str]]:
+    """
+    Automatically bind-mount directories that match MOUNTED_DIRECTORIES at build-time.
+    These directories are mounted in the "mnt-{name}" layer.
+    """
+    name = os.path.basename(path)
+    for image_name, regex in hooks.Filters.MOUNTED_DIRECTORIES.iterate():
+        if re.match(regex, name):
+            volumes.append((image_name, f"mnt-{name}"))
+    return volumes
+
+
+@hooks.Filters.COMPOSE_MOUNTS.add()
+def _mount_edx_platform_python_requirements_compose(
+    volumes: list[tuple[str, str]], folder_name: str
+) -> list[tuple[str, str]]:
+    """
+    Automatically bind-mount edx-platform Python requirements at runtime.
+    """
+    for image_name, regex in hooks.Filters.MOUNTED_DIRECTORIES.iterate():
+        if re.match(regex, folder_name):
+            # Bind-mount requirement
+            # TODO this is a breaking change because we associate runtime bind-mounts to
+            # "openedx" and no longer to "lms", "cms", etc.
+            volumes.append((image_name, f"/mnt/{folder_name}"))
+    return volumes
+
+
+def iter_mounted_directories(mounts: list[str], image_name: str) -> t.Iterator[str]:
+    """
+    Parse the list of mounted directories and yield the directory names that are for
+    the request image. Returned names are sorted in alphabetical order.
+    """
+    mounted_dirnames: set[str] = set()
+    for mount in mounts:
+        for _service, host_path, _container_path in bindmount.parse_mount(mount):
+            dirname = os.path.basename(host_path)
+            if is_directory_mounted(image_name, dirname):
+                mounted_dirnames.add(dirname)
+                break
+
+    yield from sorted(mounted_dirnames)
+
+
+def is_directory_mounted(image_name: str, dirname: str) -> bool:
+    for name, regex in hooks.Filters.MOUNTED_DIRECTORIES.iterate():
+        if name == image_name and re.match(regex, dirname):
+            return True
+    return False
+
+
+hooks.Filters.ENV_TEMPLATE_VARIABLES.add_item(
+    ("iter_mounted_directories", iter_mounted_directories)
+)

tutor/templates/build/openedx/Dockerfile

@@ -62,6 +62,11 @@ RUN git config --global user.email "tutor@overhang.io" \
 FROM scratch as edx-platform
 COPY --from=code /openedx/edx-platform /
 
+{# Create empty layers for all bind-mounted directories #}
+{% for name in iter_mounted_directories(MOUNTS, "openedx") %}
+FROM scratch as mnt-{{ name }}
+{% endfor %}
+
 ###### Download extra locales to /openedx/locale/contrib/locale
 FROM minimal as locales
 ARG OPENEDX_I18N_VERSION={{ OPENEDX_COMMON_VERSION }}
@@ -163,6 +168,7 @@ COPY --chown=app:app --from=locales /openedx/locale /openedx/locale
 COPY --chown=app:app --from=python /opt/pyenv /opt/pyenv
 COPY --chown=app:app --from=python-requirements /openedx/venv /openedx/venv
 COPY --chown=app:app --from=python-requirements /openedx/requirements /openedx/requirements
+COPY --chown=app:app --from=python-requirements /mnt /mnt
 COPY --chown=app:app --from=nodejs-requirements /openedx/nodeenv /openedx/nodeenv
 COPY --chown=app:app --from=nodejs-requirements /openedx/edx-platform/node_modules /openedx/node_modules
 
@@ -173,6 +179,12 @@ ENV PATH /openedx/venv/bin:./node_modules/.bin:/openedx/nodeenv/bin:${PATH}
 ENV VIRTUAL_ENV /openedx/venv/
 WORKDIR /openedx/edx-platform
 
+{# Install auto-mounted directories as Python packages. #}
+{% for name in iter_mounted_directories(MOUNTS, "openedx") %}
+COPY --from=mnt-{{ name }} --chown=app:app / /mnt/{{ name }}
+RUN pip install -e "/mnt/{{ name }}"
+{% endfor %}
+
 # We install edx-platform here because it creates an egg-info folder in the current
 # repo. We need both the source code and the virtualenv to run this command.
 RUN pip install -e .
@@ -265,6 +277,12 @@ RUN --mount=type=cache,target=/openedx/.cache/pip,sharing=shared \
 RUN --mount=type=cache,target=/openedx/.cache/pip,sharing=shared \
     pip install ipdb==0.13.13 ipython==8.12.0
 
+{# Re-install mounted requirements, otherwise they will be superseded by upstream reqs #}
+{% for name in iter_mounted_directories(MOUNTS, "openedx") %}
+COPY --from=mnt-{{ name }} --chown=app:app / /mnt/{{ name }}
+RUN pip install -e "/mnt/{{ name }}"
+{% endfor %}
+
 # Add ipdb as default PYTHONBREAKPOINT
 ENV PYTHONBREAKPOINT=ipdb.set_trace
 

tutor/templates/local/docker-compose.jobs.yml

@@ -27,7 +27,7 @@ services:
         - ../apps/openedx/settings/lms:/openedx/edx-platform/lms/envs/tutor:ro
         - ../apps/openedx/settings/cms:/openedx/edx-platform/cms/envs/tutor:ro
         - ../apps/openedx/config:/openedx/config:ro
-        {%- for mount in iter_mounts(MOUNTS, "lms-job") %}
+        {%- for mount in iter_mounts(MOUNTS, "openedx", "lms-job") %}
         - {{ mount }}
         {%- endfor %}
       depends_on: {{ [("mysql", RUN_MYSQL), ("mongodb", RUN_MONGODB)]|list_if }}
@@ -41,7 +41,7 @@ services:
         - ../apps/openedx/settings/lms:/openedx/edx-platform/lms/envs/tutor:ro
         - ../apps/openedx/settings/cms:/openedx/edx-platform/cms/envs/tutor:ro
         - ../apps/openedx/config:/openedx/config:ro
-        {%- for mount in iter_mounts(MOUNTS, "cms-job") %}
+        {%- for mount in iter_mounts(MOUNTS, "openedx", "cms-job") %}
         - {{ mount }}
         {%- endfor %}
       depends_on: {{ [("mysql", RUN_MYSQL), ("mongodb", RUN_MONGODB), ("elasticsearch", RUN_ELASTICSEARCH), ("redis", RUN_REDIS)]|list_if }}

tutor/templates/local/docker-compose.yml

@@ -112,7 +112,7 @@ services:
       - ../apps/openedx/uwsgi.ini:/openedx/edx-platform/uwsgi.ini:ro
       - ../../data/lms:/openedx/data
       - ../../data/openedx-media:/openedx/media
-      {%- for mount in iter_mounts(MOUNTS, "lms") %}
+      {%- for mount in iter_mounts(MOUNTS, "openedx", "lms") %}
       - {{ mount }}
       {%- endfor %}
     depends_on:
@@ -138,7 +138,7 @@ services:
       - ../apps/openedx/uwsgi.ini:/openedx/edx-platform/uwsgi.ini:ro
       - ../../data/cms:/openedx/data
       - ../../data/openedx-media:/openedx/media
-      {%- for mount in iter_mounts(MOUNTS, "cms") %}
+      {%- for mount in iter_mounts(MOUNTS, "openedx", "cms") %}
       - {{ mount }}
       {%- endfor %}
     depends_on:
@@ -166,7 +166,7 @@ services:
       - ../apps/openedx/config:/openedx/config:ro
       - ../../data/lms:/openedx/data
       - ../../data/openedx-media:/openedx/media
-      {%- for mount in iter_mounts(MOUNTS, "lms-worker") %}
+      {%- for mount in iter_mounts(MOUNTS, "openedx", "lms-worker") %}
       - {{ mount }}
       {%- endfor %}
     depends_on:
@@ -185,7 +185,7 @@ services:
       - ../apps/openedx/config:/openedx/config:ro
       - ../../data/cms:/openedx/data
       - ../../data/openedx-media:/openedx/media
-      {%- for mount in iter_mounts(MOUNTS, "cms-worker") %}
+      {%- for mount in iter_mounts(MOUNTS, "openedx", "cms-worker") %}
       - {{ mount }}
       {%- endfor %}
     depends_on: