christianlight/tutor
6
0
Fork 0
mirror of https://github.com/ChristianLight/tutor.git synced 2024-11-05 21:07:50 +00:00
Code Issues Releases Activity

feat: persistent bind-mounts

Browse Source 
This is an important change, where we get remove the previous `--mount`
option, and instead opt for persistent bind-mounts.

Persistent bind mounts have several advantages:
- They make it easier to remember which folders need to be bind-mounted.
- Code is *much* less clunky, as we no longer need to generate temporary
  docker-compose files.
- They allow us to bind-mount host directories *at build time* using the
  buildx `--build-context` option.
- The transition from development to production becomes much easier, as
  images will automatically be built using the host repo.

The only drawback is that persistent bind-mounts are slightly less
portable: when a config.yml file is moved to a different folder, many
things will break if the repo is not checked out in the same path.

For instance, this is how to start working on a local fork of
edx-platform:

    tutor config save --append MOUNTS=/path/to/edx-platform

And that's all there is to it. No, this fork will be used whenever we
run:

    tutor images build openedx
    tutor local start
    tutor dev start

This change is made possible by huge improvements in the build time
performance. These improvements make it convenient to re-build Docker
images often.

Related issues:
https://github.com/openedx/wg-developer-experience/issues/71
https://github.com/openedx/wg-developer-experience/issues/66
https://github.com/openedx/wg-developer-experience/issues/166
This commit is contained in:
Régis Behmo 2023-04-27 20:25:20 +02:00
parent 7972a75915
commit 18ce1f2fe4
17 changed files with 310 additions and 474 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
changelog.d/20230427_165520_regis_build_mount.md
View File

@ -1 +1,3 @@
- [Improvement] Automatically pull Docker image cache from the remote registry. Again, this will considerably improve image build-time, particularly in "cold-start" scenarios, where the images need to be built from scratch. The registry cache can be disabled with the `tutor images build --no-registry-cache` option. (by @regisb)
- [Feature] Automatically mount host folders *at build time*. This is a really important feature, as it allows us to transparently build images using local forks of remote repositories. (by @regisb)
- 💥[Deprecation] Remove the various `--mount` options. These options are replaced by persistent mounts. (by @regisb)

100
docs/dev.rst
View File

@ -12,31 +12,25 @@ First-time setup
Firstly, either :ref:`install Tutor <install>` (for development against the named releases of Open edX) or :ref:`install Tutor Nightly <nightly>` (for development against Open edX's master branches).
Then, optionally, tell Tutor to use a local fork of edx-platform. In that case you will need to rebuild the "openedx" Docker image::
tutor config save --append MOUNTS=./edx-platform
tutor images build openedx
Then, run one of the following in order to launch the developer platform setup process::
# To use the edx-platform repository that is built into the image, run:
tutor dev launch
# To bind-mount and run a local clone of edx-platform, replace
# './edx-platform' with the path to the local clone and run:
tutor dev launch --mount=./edx-platform
This will perform several tasks. It will:
* stop any existing locally-running Tutor containers,
* disable HTTPS,
* set ``LMS_HOST`` to `local.overhang.io <http://local.overhang.io>`_ (a convenience domain that simply `points at 127.0.0.1 <https://dnschecker.org/#A/local.overhang.io>`_),
* prompt for a platform details (with suitable defaults),
* build an ``openedx-dev`` image, which is based ``openedx`` production image but is `specialized for developer usage`_,
* start LMS, CMS, supporting services, and any plugged-in services,
* ensure databases are created and migrated, and
* run service initialization scripts, such as service user creation and Waffle configuration.
Additionally, when a local clone of edx-platform is bind-mounted, it will:
@ -55,10 +49,13 @@ Now, use the ``tutor dev ...`` command-line interface to manage the development
.. note::
Wherever the ``[--mount=./edx-platform]`` option is present, either:
If you've added your edx-platform to the ``MOUNTS`` setting, you can remove at any time by running::
* omit it when running of the edx-platform repository built into the image, or
* substitute it with ``--mount=<path/to/edx-platform>``.
tutor config save --remove MOUNTS=./edx-platform
At any time, check your configuration by running::
tutor config printvalue MOUNTS
Read more about bind-mounts :ref:`below <bind_mounts>`.
@ -74,17 +71,17 @@ Starting the platform back up
Once first-time setup has been performed with ``launch``, the platform can be started going forward with the lighter-weight ``start -d`` command, which brings up containers *detached* (that is: in the background), but does not perform any initialization tasks::
tutor dev start -d [--mount=./edx-platform]
tutor dev start -d
Or, to start with platform with containers *attached* (that is: in the foreground, the current terminal), omit the ``-d`` flag::
tutor dev start [--mount=./edx-platform]
tutor dev start
When running containers attached, stop the platform with ``Ctrl+c``, or switch to detached mode using ``Ctrl+z``.
Finally, the platform can also be started back up with ``launch``. It will take longer than ``start``, but it will ensure that config is applied, databases are provisioned & migrated, plugins are fully initialized, and (if applicable) the bind-mounted edx-platform is set up. Notably, ``launch`` is idempotent, so it is always safe to run it again without risk to data. Including the ``--pullimages`` flag will also ensure that container images are up-to-date::
tutor dev launch [--mount=./edx-platform] --pullimages
tutor dev launch --pullimages
Debugging with breakpoints
--------------------------
@ -92,32 +89,32 @@ Debugging with breakpoints
To debug a local edx-platform repository, add a `python breakpoint <https://docs.python.org/3/library/functions.html#breakpoint>`__ with ``breakpoint()`` anywhere in the code. Then, attach to the applicable service's container by running ``start`` (without ``-d``) followed by the service's name::
# Debugging LMS:
tutor dev start [--mount=./edx-platform] lms
tutor dev start lms
# Or, debugging CMS:
tutor dev start [--mount=./edx-platform] cms
tutor dev start cms
Running arbitrary commands
--------------------------
To run any command inside one of the containers, run ``tutor dev run [OPTIONS] SERVICE [COMMAND] [ARGS]...``. For instance, to open a bash shell in the LMS or CMS containers::
tutor dev run [--mount=./edx-platform] lms bash
tutor dev run [--mount=./edx-platform] cms bash
tutor dev run lms bash
tutor dev run cms bash
To open a python shell in the LMS or CMS, run::
tutor dev run [--mount=./edx-platform] lms ./manage.py lms shell
tutor dev run [--mount=./edx-platform] cms ./manage.py cms shell
tutor dev run lms ./manage.py lms shell
tutor dev run cms ./manage.py cms shell
You can then import edx-platform and django modules and execute python code.
To rebuild assets, you can use the ``openedx-assets`` command that ships with Tutor::
tutor dev run [--mount=./edx-platform] lms openedx-assets build --env=dev
tutor dev run lms openedx-assets build --env=dev
.. _specialized for developer usage:
.. _specialized for developer usage:
Rebuilding the openedx-dev image
--------------------------------
@ -143,35 +140,42 @@ Sharing directories with containers
It may sometimes be convenient to mount container directories on the host, for instance: for editing and debugging. Tutor provides different solutions to this problem.
.. _mount_option:
.. _persistent_mounts:
Bind-mount volumes with ``--mount``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Persistent bind-mounted volumes with ``MOUNTS``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The ``launch``, ``run``, ``init`` and ``start`` subcommands of ``tutor dev`` and ``tutor local`` support the ``-m/--mount`` option (see :option:`tutor dev start -m`) which can take two different forms. The first is explicit::
``MOUNTS`` is a Tutor setting to bind-mount host directories both at build time and run time:
tutor dev start --mount=lms:/path/to/edx-platform:/openedx/edx-platform lms
- At build time: plugins can automatically add certain directories listed in this setting to the `Docker build context <https://docs.docker.com/engine/reference/commandline/buildx_build/#build-context>`__. This makes it possible to transparently build a Docker image using a locally checked-out repository.
- At run time: host directories will be bind-mounted in running containers, using either an automatic or a manual configuration.
And the second is implicit::
After some values have been added to the ``MOUNTS`` setting, all ``tutor dev`` and ``tutor local`` commands will make use of these bind-mount volumes.
tutor dev start --mount=/path/to/edx-platform lms
Values added to ``MOUNTS`` can take one of two forms. The first is explicit::
With the explicit form, the ``--mount`` option means "bind-mount the host folder /path/to/edx-platform to /openedx/edx-platform in the lms container".
tutor config save --append MOUNTS=lms:/path/to/edx-platform:/openedx/edx-platform
If you use the explicit format, you will quickly realise that you usually want to bind-mount folders in multiple containers at a time. For instance, you will want to bind-mount the edx-platform repository in the "cms" container. To do that, write instead::
The second is implicit::
tutor dev start --mount=lms,cms:/path/to/edx-platform:/openedx/edx-platform lms
tutor config save --append MOUNTS=/path/to/edx-platform
This command line can become cumbersome and inconvenient to work with. But Tutor can be smart about bind-mounting folders to the right containers in the right place when you use the implicit form of the ``--mount`` option. For instance, the following commands are equivalent::
With the explicit form, the setting means "bind-mount the host folder /path/to/edx-platform to /openedx/edx-platform in the lms container at run time".
# Explicit form
tutor dev start --mount=lms,lms-worker,lms-job,cms,cms-worker,cms-job:/path/to/edx-platform:/openedx/edx-platform lms
# Implicit form
tutor dev start --mount=/path/to/edx-platform lms
If you use the explicit format, you will quickly realise that you usually want to bind-mount folders in multiple containers at a time. For instance, you will want to bind-mount the edx-platform repository in the "cms" container, but also the "lms-worker" and "cms-worker" containers. To do that, write instead::
So, when should you *not* be using the implicit form? That would be when Tutor does not know where to bind-mount your host folders. For instance, if you wanted to bind-mount your edx-platform virtual environment located in ``~/venvs/edx-platform``, you should not write ``--mount=~/venvs/edx-platform``, because that folder would be mounted in a way that would override the edx-platform repository in the container. Instead, you should write::
# each service is added to a coma-separated list
tutor config save --append MOUNTS=lms,cms,lms-worker,cms-worker:/path/to/edx-platform:/openedx/edx-platform
tutor dev start --mount=lms:~/venvs/edx-platform:/openedx/venv lms
This command line is a bit cumbersome. In addition, with this explicit form, the edx-platform repository will *not* be added to the build context at build time. But Tutor can be smart about bind-mounting folders to the right containers in the right place when you use the implicit form of the ``MOUNTS`` setting. For instance, the following implicit form can be used instead of the explicit form above::
tutor config save --append MOUNTS=/path/to/edx-platform
With this implicit form, the edx-platform repo will be bind-mounted in the containers at run time, just like with the explicit form. But in addition, the edx-platform will also automatically be added to the Docker image at build time.
So, when should you *not* be using the implicit form? That would be when Tutor does not know where to bind-mount your host folders. For instance, if you wanted to bind-mount your edx-platform virtual environment located in ``~/venvs/edx-platform``, you should not write ``--append MOUNTS=~/venvs/edx-platform``, because that folder would be mounted in a way that would override the edx-platform repository in the container. Instead, you should write::
tutor config save --append MOUNTS=lms:~/venvs/edx-platform:/openedx/venv
.. note:: Remember to setup your edx-platform repository for development! See :ref:`edx_platform_dev_env`.
@ -182,16 +186,16 @@ Sometimes, you may want to modify some of the files inside a container for which
tutor dev copyfrom lms /openedx/venv ~
Then, bind-mount that folder back in the container with the ``--mount`` option (described :ref:`above <mount_option>`)::
Then, bind-mount that folder back in the container with the ``MOUNTS`` setting (described :ref:`above <persistent_mounts>`)::
tutor dev start --mount lms:~/venv:/openedx/venv lms
tutor config save --append MOUNTS=lms:~/venv:/openedx/venv
You can then edit the files in ``~/venv`` on your local filesystem and see the changes live in your container.
You can then edit the files in ``~/venv`` on your local filesystem and see the changes live in your "lms" container.
Manual bind-mount to any directory
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. warning:: Manually bind-mounting volumes with the ``--volume`` option makes it difficult to simultaneously bind-mount to multiple containers. Also, the ``--volume`` options are not compatible with ``start`` commands. For an alternative, see the :ref:`mount option <mount_option>`.
.. warning:: Manually bind-mounting volumes with the ``--volume`` option makes it difficult to simultaneously bind-mount to multiple containers. Also, the ``--volume`` options are not compatible with ``start`` commands. For an alternative, see the :ref:`persistent mounts <persistent_mounts>`.
The above solution may not work for you if you already have an existing directory, outside of the "volumes/" directory, which you would like mounted in one of your containers. For instance, you may want to mount your copy of the `edx-platform <https://github.com/openedx/edx-platform/>`__ repository. In such cases, you can simply use the ``-v/--volume`` `Docker option <https://docs.docker.com/storage/volumes/#choose-the--v-or---mount-flag>`__::
@ -200,7 +204,7 @@ The above solution may not work for you if you already have an existing director
Override docker-compose volumes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The above solutions require that you explicitly pass the ``-m/--mount`` options to every ``run``, ``start`` or ``init`` command, which may be inconvenient. To address these issues, you can create a ``docker-compose.override.yml`` file that will specify custom volumes to be used with all ``dev`` commands::
Adding items to the ``MOUNTS`` setting effectively adds new bind-mount volumes to the ``docker-compose.yml`` files. But you might want to have more control over your volumes, such as adding read-only options, or customising other fields of the different services. To address these issues, you can create a ``docker-compose.override.yml`` file that will specify custom volumes to be used with all ``dev`` commands::
vim "$(tutor config printroot)/env/dev/docker-compose.override.yml"
@ -221,7 +225,7 @@ You are then free to bind-mount any directory to any container. For instance, to
volumes:
- /path/to/edx-platform:/openedx/edx-platform
This override file will be loaded when running any ``tutor dev ..`` command. The edx-platform repo mounted at the specified path will be automatically mounted inside all LMS and CMS containers. With this file, you should no longer specify the ``-m/--mount`` option from the command line.
This override file will be loaded when running any ``tutor dev ..`` command. The edx-platform repo mounted at the specified path will be automatically mounted inside all LMS and CMS containers.
.. note::
The ``tutor local`` commands load the ``docker-compose.override.yml`` file from the ``$(tutor config printroot)/env/local/docker-compose.override.yml`` directory. One-time jobs from initialisation commands load the ``local/docker-compose.jobs.override.yml`` and ``dev/docker-compose.jobs.override.yml``.

1
docs/tutorials/arm64.rst
View File

@ -27,6 +27,7 @@ Then, build the "openedx" and "permissions" images::
tutor images build openedx permissions
.. TODO we don't want this instruction anymore
If you want to use Tutor as an Open edX development environment, you should also build the development images::
tutor dev dc build lms

88
tests/commands/test_compose.py
View File

@ -1,88 +0,0 @@
from __future__ import annotations
import typing as t
import unittest
from io import StringIO
from unittest.mock import patch
from click.exceptions import ClickException
from tutor import hooks
from tutor.commands import compose
from tutor.commands.local import LocalContext
class ComposeTests(unittest.TestCase):
maxDiff = None # Ensure we can see long diffs of YAML files.
def test_mount_option_parsing(self) -> None:
param = compose.MountParam()
self.assertEqual(
[("lms", "/path/to/edx-platform", "/openedx/edx-platform")],
param("lms:/path/to/edx-platform:/openedx/edx-platform"),
)
self.assertEqual(
[
("lms", "/path/to/edx-platform", "/openedx/edx-platform"),
("cms", "/path/to/edx-platform", "/openedx/edx-platform"),
],
param("lms,cms:/path/to/edx-platform:/openedx/edx-platform"),
)
self.assertEqual(
[
("lms", "/path/to/edx-platform", "/openedx/edx-platform"),
("cms", "/path/to/edx-platform", "/openedx/edx-platform"),
],
param("lms, cms:/path/to/edx-platform:/openedx/edx-platform"),
)
self.assertEqual(
[
("lms", "/path/to/edx-platform", "/openedx/edx-platform"),
("lms-worker", "/path/to/edx-platform", "/openedx/edx-platform"),
],
param("lms,lms-worker:/path/to/edx-platform:/openedx/edx-platform"),
)
with self.assertRaises(ClickException):
param("lms,:/path/to/edx-platform:/openedx/edx-platform")
@patch("sys.stdout", new_callable=StringIO)
def test_compose_local_tmp_generation(self, _mock_stdout: StringIO) -> None:
"""
Ensure that docker-compose.tmp.yml is correctly generated.
"""
param = compose.MountParam()
mount_args = (
# Auto-mounting of edx-platform to lms* and cms*
param.convert_implicit_form("/path/to/edx-platform"),
# Manual mounting of some other folder to mfe and lms
param.convert_explicit_form(
"mfe,lms:/path/to/something-else:/openedx/something-else"
),
)
# Mount volumes
compose.mount_tmp_volumes(mount_args, LocalContext(""))
compose_file: dict[str, t.Any] = hooks.Filters.COMPOSE_LOCAL_TMP.apply({})
actual_services: dict[str, t.Any] = compose_file["services"]
expected_services: dict[str, t.Any] = {
"cms": {"volumes": ["/path/to/edx-platform:/openedx/edx-platform"]},
"cms-worker": {"volumes": ["/path/to/edx-platform:/openedx/edx-platform"]},
"lms": {
"volumes": [
"/path/to/edx-platform:/openedx/edx-platform",
"/path/to/something-else:/openedx/something-else",
]
},
"lms-worker": {"volumes": ["/path/to/edx-platform:/openedx/edx-platform"]},
"mfe": {"volumes": ["/path/to/something-else:/openedx/something-else"]},
}
self.assertEqual(actual_services, expected_services)
compose_jobs_file = hooks.Filters.COMPOSE_LOCAL_JOBS_TMP.apply({})
actual_jobs_services = compose_jobs_file["services"]
expected_jobs_services: dict[str, t.Any] = {
"cms-job": {"volumes": ["/path/to/edx-platform:/openedx/edx-platform"]},
"lms-job": {"volumes": ["/path/to/edx-platform:/openedx/edx-platform"]},
}
self.assertEqual(actual_jobs_services, expected_jobs_services)

2
tests/commands/test_images.py
View File

@ -149,7 +149,7 @@ class ImagesTests(PluginsTestCase, TestCommandMixin):
"docker_args",
"--cache-from=type=registry,ref=service1:1.0.0-cache",
],
list(image_build.call_args[0][1:])
list(image_build.call_args[0][1:]),
)
def test_images_push(self) -> None:

65
tests/test_bindmount.py Normal file
View File

@ -0,0 +1,65 @@
from __future__ import annotations
import unittest
from tutor import bindmount
class BindmountTests(unittest.TestCase):
def test_parse_explicit(self) -> None:
self.assertEqual(
[("lms", "/path/to/edx-platform", "/openedx/edx-platform")],
bindmount.parse_explicit_mount(
"lms:/path/to/edx-platform:/openedx/edx-platform"
),
)
self.assertEqual(
[
("lms", "/path/to/edx-platform", "/openedx/edx-platform"),
("cms", "/path/to/edx-platform", "/openedx/edx-platform"),
],
bindmount.parse_explicit_mount(
"lms,cms:/path/to/edx-platform:/openedx/edx-platform"
),
)
self.assertEqual(
[
("lms", "/path/to/edx-platform", "/openedx/edx-platform"),
("cms", "/path/to/edx-platform", "/openedx/edx-platform"),
],
bindmount.parse_explicit_mount(
"lms, cms:/path/to/edx-platform:/openedx/edx-platform"
),
)
self.assertEqual(
[
("lms", "/path/to/edx-platform", "/openedx/edx-platform"),
("lms-worker", "/path/to/edx-platform", "/openedx/edx-platform"),
],
bindmount.parse_explicit_mount(
"lms,lms-worker:/path/to/edx-platform:/openedx/edx-platform"
),
)
self.assertEqual(
[("lms", "/path/to/edx-platform", "/openedx/edx-platform")],
bindmount.parse_explicit_mount(
"lms,:/path/to/edx-platform:/openedx/edx-platform"
),
)
def test_parse_implicit(self) -> None:
# Import module to make sure filter is created
# pylint: disable=import-outside-toplevel,unused-import
import tutor.commands.compose
self.assertEqual(
[
("lms", "/path/to/edx-platform", "/openedx/edx-platform"),
("cms", "/path/to/edx-platform", "/openedx/edx-platform"),
("lms-worker", "/path/to/edx-platform", "/openedx/edx-platform"),
("cms-worker", "/path/to/edx-platform", "/openedx/edx-platform"),
("lms-job", "/path/to/edx-platform", "/openedx/edx-platform"),
("cms-job", "/path/to/edx-platform", "/openedx/edx-platform"),
],
bindmount.parse_implicit_mount("/path/to/edx-platform"),
)

71
tutor/bindmount.py Normal file
View File

@ -0,0 +1,71 @@
from __future__ import annotations
from functools import lru_cache
import os
import re
import typing as t
from tutor import hooks
def iter_mounts(user_mounts: list[str], name: str) -> t.Iterable[str]:
"""
Iterate on the bind-mounts that are available to any given compose service. The list
of bind-mounts is parsed from `user_mounts` and we yield only those for service
`name`.
Calling this function multiple times makes repeated calls to the parsing functions,
but that's OK because their result is cached.
"""
for user_mount in user_mounts:
for service, host_path, container_path in parse_mount(user_mount):
if service == name:
yield f"{host_path}:{container_path}"
def parse_mount(value: str) -> list[tuple[str, str, str]]:
"""
Parser for mount arguments of the form "service1[,service2,...]:/host/path:/container/path".
Returns a list of (service, host_path, container_path) tuples.
"""
mounts = parse_explicit_mount(value) or parse_implicit_mount(value)
return mounts
@lru_cache(maxsize=None)
def parse_explicit_mount(value: str) -> list[tuple[str, str, str]]:
"""
Argument is of the form "containers:/host/path:/container/path".
"""
# Note that this syntax does not allow us to include colon ':' characters in paths
match = re.match(
r"(?P<services>[a-zA-Z0-9-_, ]+):(?P<host_path>[^:]+):(?P<container_path>[^:]+)",
value,
)
if not match:
return []
mounts: list[tuple[str, str, str]] = []
services: list[str] = [service.strip() for service in match["services"].split(",")]
host_path = os.path.abspath(os.path.expanduser(match["host_path"]))
host_path = host_path.replace(os.path.sep, "/")
container_path = match["container_path"]
for service in services:
if service:
mounts.append((service, host_path, container_path))
return mounts
@lru_cache(maxsize=None)
def parse_implicit_mount(value: str) -> list[tuple[str, str, str]]:
"""
Argument is of the form "/host/path"
"""
mounts: list[tuple[str, str, str]] = []
host_path = os.path.abspath(os.path.expanduser(value))
for service, container_path in hooks.Filters.COMPOSE_MOUNTS.iterate(
os.path.basename(host_path)
):
mounts.append((service, host_path, container_path))
return mounts

61
tutor/bindmounts.py
View File

@ -1,61 +0,0 @@
import os
from tutor.exceptions import TutorError
from tutor.tasks import BaseComposeTaskRunner
from tutor.utils import get_user_id
def create(
runner: BaseComposeTaskRunner,
service: str,
path: str,
) -> str:
volumes_root_path = get_root_path(runner.root)
volume_name = get_name(path)
container_volumes_root_path = "/tmp/volumes"
command = """rm -rf {volumes_path}/{volume_name}
cp -r {src_path} {volumes_path}/{volume_name}
chown -R {user_id} {volumes_path}/{volume_name}""".format(
volumes_path=container_volumes_root_path,
volume_name=volume_name,
src_path=path,
user_id=get_user_id(),
)
# Create volumes root dir if it does not exist. Otherwise it is created with root owner and might not be writable
# in the container, e.g: in the dev containers.
if not os.path.exists(volumes_root_path):
os.makedirs(volumes_root_path)
runner.docker_compose(
"run",
"--rm",
"--no-deps",
"--user=0",
"--volume",
f"{volumes_root_path}:{container_volumes_root_path}",
service,
"sh",
"-e",
"-c",
command,
)
return os.path.join(volumes_root_path, volume_name)
def get_path(root: str, container_bind_path: str) -> str:
bind_basename = get_name(container_bind_path)
return os.path.join(get_root_path(root), bind_basename)
def get_name(container_bind_path: str) -> str:
# We rstrip slashes, otherwise os.path.basename returns an empty string
# We don't use basename here as it will not work on Windows
name = container_bind_path.rstrip("/").split("/")[-1]
if not name:
raise TutorError("Mounting a container root folder is not supported")
return name
def get_root_path(root: str) -> str:
return os.path.join(root, "volumes")

203
tutor/commands/compose.py
View File

@ -1,17 +1,12 @@
from __future__ import annotations
import os
import re
import typing as t
from copy import deepcopy
import click
from click.shell_completion import CompletionItem
from typing_extensions import TypeAlias
from tutor import config as tutor_config
from tutor import env as tutor_env
from tutor import fmt, hooks, serialize, utils
from tutor import bindmount, hooks, utils
from tutor.commands import jobs
from tutor.commands.context import BaseTaskContext
from tutor.core.hooks import Filter # pylint: disable=unused-import
@ -19,8 +14,6 @@ from tutor.exceptions import TutorError
from tutor.tasks import BaseComposeTaskRunner
from tutor.types import Config
COMPOSE_FILTER_TYPE: TypeAlias = "Filter[dict[str, t.Any], []]"
class ComposeTaskRunner(BaseComposeTaskRunner):
def __init__(self, root: str, config: Config):
@ -47,47 +40,6 @@ class ComposeTaskRunner(BaseComposeTaskRunner):
*args, "--project-name", self.project_name, *command
)
def update_docker_compose_tmp(
self,
compose_tmp_filter: COMPOSE_FILTER_TYPE,
compose_jobs_tmp_filter: COMPOSE_FILTER_TYPE,
docker_compose_tmp_path: str,
docker_compose_jobs_tmp_path: str,
) -> None:
"""
Update the contents of the docker-compose.tmp.yml and
docker-compose.jobs.tmp.yml files, which are generated at runtime.
"""
compose_base: dict[str, t.Any] = {
"version": "{{ DOCKER_COMPOSE_VERSION }}",
"services": {},
}
# 1. Apply compose_tmp filter
# 2. Render the resulting dict
# 3. Serialize to yaml
# 4. Save to disk
docker_compose_tmp: str = serialize.dumps(
tutor_env.render_unknown(
self.config, compose_tmp_filter.apply(deepcopy(compose_base))
)
)
tutor_env.write_to(
docker_compose_tmp,
docker_compose_tmp_path,
)
# Same thing but with tmp jobs
docker_compose_jobs_tmp: str = serialize.dumps(
tutor_env.render_unknown(
self.config, compose_jobs_tmp_filter.apply(deepcopy(compose_base))
)
)
tutor_env.write_to(
docker_compose_jobs_tmp,
docker_compose_jobs_tmp_path,
)
def run_task(self, service: str, command: str) -> int:
"""
Run the "{{ service }}-job" service from local/docker-compose.jobs.yml with the
@ -113,148 +65,22 @@ class ComposeTaskRunner(BaseComposeTaskRunner):
class BaseComposeContext(BaseTaskContext):
COMPOSE_TMP_FILTER: COMPOSE_FILTER_TYPE = NotImplemented
COMPOSE_JOBS_TMP_FILTER: COMPOSE_FILTER_TYPE = NotImplemented
def job_runner(self, config: Config) -> ComposeTaskRunner:
raise NotImplementedError
class MountParam(click.ParamType):
"""
Parser for --mount arguments of the form "service1[,service2,...]:/host/path:/container/path".
"""
name = "mount"
MountType = t.Tuple[str, str, str]
# Note that this syntax does not allow us to include colon ':' characters in paths
PARAM_REGEXP = (
r"(?P<services>[a-zA-Z0-9-_, ]+):(?P<host_path>[^:]+):(?P<container_path>[^:]+)"
)
def convert(
self,
value: str,
param: t.Optional["click.Parameter"],
ctx: t.Optional[click.Context],
) -> list["MountType"]:
mounts = self.convert_explicit_form(value) or self.convert_implicit_form(value)
return mounts
def convert_explicit_form(self, value: str) -> list["MountParam.MountType"]:
"""
Argument is of the form "containers:/host/path:/container/path".
"""
match = re.match(self.PARAM_REGEXP, value)
if not match:
return []
mounts: list["MountParam.MountType"] = []
services: list[str] = [
service.strip() for service in match["services"].split(",")
]
host_path = os.path.abspath(os.path.expanduser(match["host_path"]))
host_path = host_path.replace(os.path.sep, "/")
container_path = match["container_path"]
for service in services:
if not service:
self.fail(f"incorrect services syntax: '{match['services']}'")
mounts.append((service, host_path, container_path))
return mounts
def convert_implicit_form(self, value: str) -> list["MountParam.MountType"]:
"""
Argument is of the form "/host/path"
"""
mounts: list["MountParam.MountType"] = []
host_path = os.path.abspath(os.path.expanduser(value))
for service, container_path in hooks.Filters.COMPOSE_MOUNTS.iterate(
os.path.basename(host_path)
):
mounts.append((service, host_path, container_path))
if not mounts:
raise self.fail(f"no mount found for {value}")
return mounts
def shell_complete(
self, ctx: click.Context, param: click.Parameter, incomplete: str
) -> list[CompletionItem]:
"""
Mount argument completion works only for the single path (implicit) form. The
reason is that colons break words in bash completion:
http://tiswww.case.edu/php/chet/bash/FAQ (E13)
Thus, we do not even attempt to auto-complete mount arguments that include
colons: such arguments will not even reach this method.
"""
return [CompletionItem(incomplete, type="file")]
mount_option = click.option(
"-m",
"--mount",
"mounts",
help="""Bind-mount a folder from the host in the right containers. This option can take two different forms. The first one is explicit: 'service1[,service2...]:/host/path:/container/path'. The other is implicit: '/host/path'. Arguments passed in the implicit form will be parsed by plugins to define the right folders to bind-mount from the host.""",
type=MountParam(),
multiple=True,
)
def mount_tmp_volumes(
all_mounts: tuple[list[MountParam.MountType], ...],
context: BaseComposeContext,
) -> None:
for mounts in all_mounts:
for service, host_path, container_path in mounts:
mount_tmp_volume(service, host_path, container_path, context)
def mount_tmp_volume(
service: str,
host_path: str,
container_path: str,
context: BaseComposeContext,
) -> None:
"""
Append user-defined bind-mounted volumes to the docker-compose.tmp file(s).
The service/host path/container path values are appended to the docker-compose
files by mean of two filters. Each dev/local environment is then responsible for
generating the files based on the output of these filters.
Bind-mounts that are associated to "*-job" services will be added to the
docker-compose jobs file.
"""
fmt.echo_info(f"Bind-mount: {host_path} -> {container_path} in {service}")
compose_tmp_filter: COMPOSE_FILTER_TYPE = (
context.COMPOSE_JOBS_TMP_FILTER
if service.endswith("-job")
else context.COMPOSE_TMP_FILTER
)
@compose_tmp_filter.add()
def _add_mounts_to_docker_compose_tmp(
docker_compose: dict[str, t.Any],
) -> dict[str, t.Any]:
services = docker_compose.setdefault("services", {})
services.setdefault(service, {"volumes": []})
services[service]["volumes"].append(f"{host_path}:{container_path}")
return docker_compose
@click.command(
short_help="Run all or a selection of services.",
help="Run all or a selection of services. Docker images will be rebuilt where necessary.",
)
@click.option("--skip-build", is_flag=True, help="Skip image building")
@click.option("-d", "--detach", is_flag=True, help="Start in daemon mode")
@mount_option
@click.argument("services", metavar="service", nargs=-1)
@click.pass_obj
def start(
context: BaseComposeContext,
skip_build: bool,
detach: bool,
mounts: tuple[list[MountParam.MountType]],
services: list[str],
) -> None:
command = ["up", "--remove-orphans"]
@ -264,7 +90,6 @@ def start(
command.append("-d")
# Start services
mount_tmp_volumes(mounts, context)
config = tutor_config.load(context.root)
context.job_runner(config).docker_compose(*command, *services)
@ -313,21 +138,11 @@ def restart(context: BaseComposeContext, services: list[str]) -> None:
@jobs.do_group
@mount_option
@click.pass_obj
def do(context: BaseComposeContext, mounts: tuple[list[MountParam.MountType]]) -> None:
def do() -> None:
"""
Run a custom job in the right container(s).
"""
@hooks.Actions.DO_JOB.add()
def _mount_tmp_volumes(_job_name: str, *_args: t.Any, **_kwargs: t.Any) -> None:
"""
We add this logic to an action callback because we do not want to trigger it
whenever we run `tutor local do <job> --help`.
"""
mount_tmp_volumes(mounts, context)
@click.command(
short_help="Run a command in a new container",
@ -338,18 +153,16 @@ def do(context: BaseComposeContext, mounts: tuple[list[MountParam.MountType]]) -
),
context_settings={"ignore_unknown_options": True},
)
@mount_option
@click.argument("args", nargs=-1, required=True)
@click.pass_context
def run(
context: click.Context,
mounts: tuple[list[MountParam.MountType]],
args: list[str],
) -> None:
extra_args = ["--rm"]
if not utils.is_a_tty():
extra_args.append("-T")
context.invoke(dc_command, mounts=mounts, command="run", args=[*extra_args, *args])
context.invoke(dc_command, command="run", args=[*extra_args, *args])
@click.command(
@ -446,17 +259,14 @@ def status(context: click.Context) -> None:
context_settings={"ignore_unknown_options": True},
name="dc",
)
@mount_option
@click.argument("command")
@click.argument("args", nargs=-1)
@click.pass_obj
def dc_command(
context: BaseComposeContext,
mounts: tuple[list[MountParam.MountType]],
command: str,
args: list[str],
) -> None:
mount_tmp_volumes(mounts, context)
config = tutor_config.load(context.root)
context.job_runner(config).docker_compose(command, *args)
@ -466,8 +276,8 @@ def _mount_edx_platform(
volumes: list[tuple[str, str]], name: str
) -> list[tuple[str, str]]:
"""
When mounting edx-platform with `--mount=/path/to/edx-platform`, bind-mount the host
repo in the lms/cms containers.
When mounting edx-platform with `tutor config save --append MOUNTS=/path/to/edx-platform`,
bind-mount the host repo in the lms/cms containers.
"""
if name == "edx-platform":
path = "/openedx/edx-platform"
@ -482,6 +292,9 @@ def _mount_edx_platform(
return volumes
hooks.Filters.ENV_TEMPLATE_VARIABLES.add_item(("iter_mounts", bindmount.iter_mounts))
def add_commands(command_group: click.Group) -> None:
command_group.add_command(start)
command_group.add_command(stop)

21
tutor/commands/dev.py
View File

@ -18,39 +18,21 @@ class DevTaskRunner(compose.ComposeTaskRunner):
"""
super().__init__(root, config)
self.project_name = get_typed(self.config, "DEV_PROJECT_NAME", str)
docker_compose_tmp_path = tutor_env.pathjoin(
self.root, "dev", "docker-compose.tmp.yml"
)
docker_compose_jobs_tmp_path = tutor_env.pathjoin(
self.root, "dev", "docker-compose.jobs.tmp.yml"
)
self.docker_compose_files += [
tutor_env.pathjoin(self.root, "local", "docker-compose.yml"),
tutor_env.pathjoin(self.root, "dev", "docker-compose.yml"),
docker_compose_tmp_path,
tutor_env.pathjoin(self.root, "local", "docker-compose.override.yml"),
tutor_env.pathjoin(self.root, "dev", "docker-compose.override.yml"),
]
self.docker_compose_job_files += [
tutor_env.pathjoin(self.root, "local", "docker-compose.jobs.yml"),
tutor_env.pathjoin(self.root, "dev", "docker-compose.jobs.yml"),
docker_compose_jobs_tmp_path,
tutor_env.pathjoin(self.root, "local", "docker-compose.jobs.override.yml"),
tutor_env.pathjoin(self.root, "dev", "docker-compose.jobs.override.yml"),
]
# Update docker-compose.tmp files
self.update_docker_compose_tmp(
hooks.Filters.COMPOSE_DEV_TMP,
hooks.Filters.COMPOSE_DEV_JOBS_TMP,
docker_compose_tmp_path,
docker_compose_jobs_tmp_path,
)
class DevContext(compose.BaseComposeContext):
COMPOSE_TMP_FILTER = hooks.Filters.COMPOSE_DEV_TMP
COMPOSE_JOBS_TMP_FILTER = hooks.Filters.COMPOSE_DEV_JOBS_TMP
def job_runner(self, config: Config) -> DevTaskRunner:
return DevTaskRunner(self.root, config)
@ -64,15 +46,12 @@ def dev(context: click.Context) -> None:
@click.command(help="Configure and run Open edX from scratch, for development")
@click.option("-I", "--non-interactive", is_flag=True, help="Run non-interactively")
@click.option("-p", "--pullimages", is_flag=True, help="Update docker images")
@compose.mount_option
@click.pass_context
def launch(
context: click.Context,
non_interactive: bool,
pullimages: bool,
mounts: tuple[list[compose.MountParam.MountType]],
) -> None:
compose.mount_tmp_volumes(mounts, context.obj)
utils.warn_macos_docker_memory()
click.echo(fmt.title("Interactive platform configuration"))

52
tutor/commands/images.py
View File

@ -1,12 +1,13 @@
from __future__ import annotations
import os
import typing as t
import click
from tutor import config as tutor_config
from tutor import env as tutor_env
from tutor import exceptions, hooks, images, utils
from tutor import exceptions, hooks, images, types, utils
from tutor.commands.context import Context
from tutor.core.hooks import Filter
from tutor.types import Config
@ -148,16 +149,26 @@ def build(
command_args.append(f"--output={docker_output}")
if docker_args:
command_args += docker_args
# Build context mounts
build_contexts = get_image_build_contexts(config)
for image in image_names:
for _name, path, tag, custom_args in find_images_to_build(config, image):
for name, path, tag, custom_args in find_images_to_build(config, image):
image_build_args = [*command_args, *custom_args]
# Registry cache
if not no_registry_cache:
# Use registry cache
image_build_args.append(f"--cache-from=type=registry,ref={tag}-cache")
if cache_to_registry:
image_build_args.append(
f"--cache-to=type=registry,mode=max,ref={tag}-cache"
)
# Build contexts
for host_path, stage_name in build_contexts.get(name, []):
image_build_args.append(f"--build-context={stage_name}={host_path}")
# Build
images.build(
tutor_env.pathjoin(context.root, *path),
tag,
@ -165,6 +176,41 @@ def build(
)
def get_image_build_contexts(config: Config) -> dict[str, list[tuple[str, str]]]:
"""
Return all build contexts for all images.
A build context is to bind-mount a host directory at build-time. This is useful, for
instance to build a Docker image with a local git checkout of a remote repo.
Users configure bind-mounts with the `MOUNTS` config setting. Plugins can then
automaticall add build contexts based on these values.
"""
user_mounts = types.get_typed(config, "MOUNTS", list)
build_contexts: dict[str, list[tuple[str, str]]] = {}
for user_mount in user_mounts:
for image_name, stage_name in hooks.Filters.IMAGES_BUILD_MOUNTS.iterate(
user_mount
):
if image_name not in build_contexts:
build_contexts[image_name] = []
build_contexts[image_name].append((user_mount, stage_name))
return build_contexts
@hooks.Filters.IMAGES_BUILD_MOUNTS.add()
def _mount_edx_platform(
volumes: list[tuple[str, str]], path: str
) -> list[tuple[str, str]]:
"""
Automatically add an edx-platform repo from the host to the build context whenever
it is added to the `MOUNTS` setting.
"""
if os.path.basename(path) == "edx-platform":
volumes.append(("openedx", "edx-platform"))
return volumes
@click.command(short_help="Pull images from the Docker registry")
@click.argument("image_names", metavar="image", nargs=-1)
@click.pass_obj

22
tutor/commands/local.py
View File

@ -23,38 +23,19 @@ class LocalTaskRunner(compose.ComposeTaskRunner):
"""
super().__init__(root, config)
self.project_name = get_typed(self.config, "LOCAL_PROJECT_NAME", str)
docker_compose_tmp_path = tutor_env.pathjoin(
self.root, "local", "docker-compose.tmp.yml"
)
docker_compose_jobs_tmp_path = tutor_env.pathjoin(
self.root, "local", "docker-compose.jobs.tmp.yml"
)
self.docker_compose_files += [
tutor_env.pathjoin(self.root, "local", "docker-compose.yml"),
tutor_env.pathjoin(self.root, "local", "docker-compose.prod.yml"),
docker_compose_tmp_path,
tutor_env.pathjoin(self.root, "local", "docker-compose.override.yml"),
]
self.docker_compose_job_files += [
tutor_env.pathjoin(self.root, "local", "docker-compose.jobs.yml"),
docker_compose_jobs_tmp_path,
tutor_env.pathjoin(self.root, "local", "docker-compose.jobs.override.yml"),
]
# Update docker-compose.tmp files
self.update_docker_compose_tmp(
hooks.Filters.COMPOSE_LOCAL_TMP,
hooks.Filters.COMPOSE_LOCAL_JOBS_TMP,
docker_compose_tmp_path,
docker_compose_jobs_tmp_path,
)
# pylint: disable=too-few-public-methods
class LocalContext(compose.BaseComposeContext):
COMPOSE_TMP_FILTER = hooks.Filters.COMPOSE_LOCAL_TMP
COMPOSE_JOBS_TMP_FILTER = hooks.Filters.COMPOSE_LOCAL_JOBS_TMP
def job_runner(self, config: Config) -> LocalTaskRunner:
return LocalTaskRunner(self.root, config)
@ -66,17 +47,14 @@ def local(context: click.Context) -> None:
@click.command(help="Configure and run Open edX from scratch")
@compose.mount_option
@click.option("-I", "--non-interactive", is_flag=True, help="Run non-interactively")
@click.option("-p", "--pullimages", is_flag=True, help="Update docker images")
@click.pass_context
def launch(
context: click.Context,
mounts: tuple[list[compose.MountParam.MountType]],
non_interactive: bool,
pullimages: bool,
) -> None:
compose.mount_tmp_volumes(mounts, context.obj)
utils.warn_macos_docker_memory()
run_upgrade_from_release = tutor_env.should_upgrade_from_release(context.obj.root)

55
tutor/hooks/catalog.py
View File

@ -227,33 +227,20 @@ class Filters:
"commands:pre-init"
)
#: Same as :py:data:`COMPOSE_LOCAL_JOBS_TMP` but for the development environment.
COMPOSE_DEV_JOBS_TMP: Filter[Config, []] = filters.get("compose:dev-jobs:tmp")
#: Same as :py:data:`COMPOSE_LOCAL_TMP` but for the development environment.
COMPOSE_DEV_TMP: Filter[Config, []] = filters.get("compose:dev:tmp")
#: Same as :py:data:`COMPOSE_LOCAL_TMP` but for jobs
COMPOSE_LOCAL_JOBS_TMP: Filter[Config, []] = filters.get("compose:local-jobs:tmp")
#: Contents of the (local|dev)/docker-compose.tmp.yml files that will be generated at
#: runtime. This is used for instance to bind-mount folders from the host (see
#: :py:data:`COMPOSE_MOUNTS`)
#:
#: :parameter dict[str, ...] docker_compose_tmp: values which will be serialized to local/docker-compose.tmp.yml.
#: Keys and values will be rendered before saving, such that you may include ``{{ ... }}`` statements.
COMPOSE_LOCAL_TMP: Filter[Config, []] = filters.get("compose:local:tmp")
#: List of folders to bind-mount in docker-compose containers, either in ``tutor local`` or ``tutor dev``.
#:
#: Many ``tutor local`` and ``tutor dev`` commands support ``--mounts`` options
#: that allow plugins to define custom behaviour at runtime. For instance
#: ``--mount=/path/to/edx-platform`` would cause this host folder to be
#: bind-mounted in different containers (lms, lms-worker, cms, cms-worker) at the
#: This filter is for processing values of the ``MOUNTS`` setting such as::
#:
#: tutor config save --append MOUNTS=/path/to/edx-platform
#:
#: In this example, this host folder would be bind-mounted in different containers
#: (lms, lms-worker, cms, cms-worker, lms-job, cms-job) at the
#: /openedx/edx-platform location. Plugin developers may implement this filter to
#: define custom behaviour when mounting folders that relate to their plugins. For
#: instance, the ecommerce plugin may process the ``--mount=/path/to/ecommerce``
#: option.
#: instance, the ecommerce plugin may process the ``/path/to/ecommerce`` value.
#:
#: To also bind-mount these folder at build time, implement also the
#: :py:data:`IMAGES_BUILD_MOUNTS` filter.
#:
#: :parameter list[tuple[str, str]] mounts: each item is a ``(service, path)``
#: tuple, where ``service`` is the name of the docker-compose service and ``path`` is
@ -262,7 +249,7 @@ class Filters:
#: the ``path`` because it will fail on Windows.
#: :parameter str name: basename of the host-mounted folder. In the example above,
#: this is "edx-platform". When implementing this filter you should check this name to
#: conditionnally add mounts.
#: conditionally add mounts.
COMPOSE_MOUNTS: Filter[list[tuple[str, str]], [str]] = filters.get("compose:mounts")
#: Declare new default configuration settings that don't necessarily have to be saved in the user
@ -402,6 +389,26 @@ class Filters:
list[tuple[str, tuple[str, ...], str, tuple[str, ...]]], [Config]
] = filters.get("images:build")
#: List of host directories to be automatically bind-mounted in Docker images at
#: build time. For instance, this is useful to build Docker images using a custom
#: repository on the host.
#:
#: This filter works similarly to the :py:data:`COMPOSE_MOUNTS` filter, with a few differences.
#:
#: :parameter list[tuple[str, str]] mounts: each item is a pair of ``(name, value)``
#: used to generate a build context at build time. See the corresponding `Docker
#: documentation <https://docs.docker.com/engine/reference/commandline/buildx_build/#build-context>`__.
#: The following option will be added to the ``docker buildx build`` command:
#: ``--build-context={name}={value}``. If the Dockerfile contains a "name" stage, then
#: that stage will be replaced by the corresponding directory on the host.
#: :parameter str name: full path to the host-mounted folder. As opposed to
#: :py:data:`COMPOSE_MOUNTS`, this is not just the basename, but the full path. When
#: implementing this filter you should check this path (for instance: with
#: ``os.path.basename(path)``) to conditionally add mounts.
IMAGES_BUILD_MOUNTS: Filter[list[tuple[str, str]], [str]] = filters.get(
"images:build:mounts"
)
#: List of images to be pulled when we run ``tutor images pull ...``.
#:
#: :parameter list[tuple[str, str]] tasks: list of ``(name, tag)`` tuples.

2
tutor/images.py
View File

@ -9,7 +9,7 @@ def get_tag(config: Config, name: str) -> str:
def build(path: str, tag: str, *args: str) -> None:
fmt.echo_info(f"Building image {tag}")
build_command = ["build", "-t", tag, *args, path]
build_command = ["build", f"--tag={tag}", *args, path]
if utils.is_buildkit_enabled():
build_command.insert(0, "buildx")
command = hooks.Filters.DOCKER_BUILD_COMMAND.apply(build_command)

1
tutor/templates/config/defaults.yml
View File

@ -43,6 +43,7 @@ MONGODB_USERNAME: ""
MONGODB_PASSWORD: ""
MONGODB_REPLICA_SET: ""
MONGODB_USE_SSL: false
MOUNTS: []
OPENEDX_AWS_ACCESS_KEY: ""
OPENEDX_AWS_SECRET_ACCESS_KEY: ""
OPENEDX_CACHE_REDIS_DB: 1

6
tutor/templates/local/docker-compose.jobs.yml
View File

@ -27,6 +27,9 @@ services:
- ../apps/openedx/settings/lms:/openedx/edx-platform/lms/envs/tutor:ro
- ../apps/openedx/settings/cms:/openedx/edx-platform/cms/envs/tutor:ro
- ../apps/openedx/config:/openedx/config:ro
{%- for mount in iter_mounts(MOUNTS, "lms-job") %}
- {{ mount }}
{%- endfor %}
depends_on: {{ [("mysql", RUN_MYSQL), ("mongodb", RUN_MONGODB)]|list_if }}
cms-job:
@ -38,6 +41,9 @@ services:
- ../apps/openedx/settings/lms:/openedx/edx-platform/lms/envs/tutor:ro
- ../apps/openedx/settings/cms:/openedx/edx-platform/cms/envs/tutor:ro
- ../apps/openedx/config:/openedx/config:ro
{%- for mount in iter_mounts(MOUNTS, "cms-job") %}
- {{ mount }}
{%- endfor %}
depends_on: {{ [("mysql", RUN_MYSQL), ("mongodb", RUN_MONGODB), ("elasticsearch", RUN_ELASTICSEARCH), ("redis", RUN_REDIS)]|list_if }}
{{ patch("local-docker-compose-jobs-services")|indent(4) }}

32
tutor/templates/local/docker-compose.yml
View File

@ -24,7 +24,7 @@ services:
############# External services
{% if RUN_MONGODB %}
{% if RUN_MONGODB -%}
mongodb:
image: {{ DOCKER_IMAGE_MONGODB }}
# Use WiredTiger in all environments, just like at edx.org
@ -35,9 +35,9 @@ services:
- ../../data/mongodb:/data/db
depends_on:
- permissions
{% endif %}
{%- endif %}
{% if RUN_MYSQL %}
{% if RUN_MYSQL -%}
mysql:
image: {{ DOCKER_IMAGE_MYSQL }}
command: mysqld --character-set-server=utf8 --collation-server=utf8_general_ci
@ -47,9 +47,9 @@ services:
- ../../data/mysql:/var/lib/mysql
environment:
MYSQL_ROOT_PASSWORD: "{{ MYSQL_ROOT_PASSWORD }}"
{% endif %}
{%- endif %}
{% if RUN_ELASTICSEARCH %}
{% if RUN_ELASTICSEARCH -%}
elasticsearch:
image: {{ DOCKER_IMAGE_ELASTICSEARCH }}
environment:
@ -67,9 +67,9 @@ services:
- ../../data/elasticsearch:/usr/share/elasticsearch/data
depends_on:
- permissions
{% endif %}
{%- endif %}
{% if RUN_REDIS %}
{% if RUN_REDIS -%}
redis:
image: {{ DOCKER_IMAGE_REDIS }}
working_dir: /openedx/redis/data
@ -81,16 +81,16 @@ services:
restart: unless-stopped
depends_on:
- permissions
{% endif %}
{%- endif %}
{% if RUN_SMTP %}
{% if RUN_SMTP -%}
smtp:
image: {{ DOCKER_IMAGE_SMTP }}
restart: unless-stopped
user: "100:101"
environment:
HOSTNAME: "{{ LMS_HOST }}"
{% endif %}
{%- endif %}
############# LMS and CMS
@ -108,6 +108,9 @@ services:
- ../apps/openedx/uwsgi.ini:/openedx/edx-platform/uwsgi.ini:ro
- ../../data/lms:/openedx/data
- ../../data/openedx-media:/openedx/media
{%- for mount in iter_mounts(MOUNTS, "lms") %}
- {{ mount }}
{%- endfor %}
depends_on:
- permissions
{% if RUN_MYSQL %}- mysql{% endif %}
@ -131,6 +134,9 @@ services:
- ../apps/openedx/uwsgi.ini:/openedx/edx-platform/uwsgi.ini:ro
- ../../data/cms:/openedx/data
- ../../data/openedx-media:/openedx/media
{%- for mount in iter_mounts(MOUNTS, "cms") %}
- {{ mount }}
{%- endfor %}
depends_on:
- permissions
- lms
@ -156,6 +162,9 @@ services:
- ../apps/openedx/config:/openedx/config:ro
- ../../data/lms:/openedx/data
- ../../data/openedx-media:/openedx/media
{%- for mount in iter_mounts(MOUNTS, "lms-worker") %}
- {{ mount }}
{%- endfor %}
depends_on:
- lms
@ -172,6 +181,9 @@ services:
- ../apps/openedx/config:/openedx/config:ro
- ../../data/cms:/openedx/data
- ../../data/openedx-media:/openedx/media
{%- for mount in iter_mounts(MOUNTS, "cms-worker") %}
- {{ mount }}
{%- endfor %}
depends_on:
- cms