christianlight/tutor
7
0
Fork 0
mirror of https://github.com/ChristianLight/tutor.git synced 2024-05-28 20:00:49 +00:00
Code Issues Releases Activity

feat: add --mount option to local/dev

Browse Source 
The `--mount` option is available both with `tutor local`
and `tutor dev` commands. It allows users to easily bind-mount containers from
the host to containers. Yes, I know, we already provide that possibility with
the `bindmount` command and the `--volume=/path/` option. But these suffer from
the following drawbacks:

- They are difficult to understand.
- The "bindmount" command name does not make much sense.
- It's not convenient to mount an arbitrary folder from the host to multiple
  containers, such as the many lms/cms containers (web apps, celery workers and
  job runners).

To address this situation, we now recommend to make use of --mount:

1. `--mount=service1[,service2,...]:/host/path:/container/path`: manually mount
   `/host/path` to `/container/path` in container "service1" (and "service2").
2. `--mount=/host/path`: use the new v1 plugin API to discover plugins that
   will detect this option and select the right containers in which to bind-mount
   volumes. This is really nifty...

Close https://github.com/overhangio/2u-tutor-adoption/issues/43
This commit is contained in:
Régis Behmo 2022-04-15 10:51:19 +02:00 committed by Régis Behmo
parent df0e26c58e
commit d9486018a2
15 changed files with 354 additions and 67 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
CHANGELOG.md
View File

@ -4,6 +4,7 @@ Note: Breaking changes between versions are indicated by "💥".
## Unreleased
- [Feature] Introduce the ``-m/--mount`` option in ``local`` and ``dev`` commands to auto-magically bind-mount folders from the host.
- [Feature] Add `tutor dev quickstart` command, which is similar to `tutor local quickstart`, except that it uses dev containers instead of local production ones and includes some other small differences for the convience of Open edX developers. This should remove some friction from the Open edX development setup process, which previously required that users provision using local producation containers (`tutor local quickstart`) but then stop them and switch to dev containers (`tutor local stop && tutor dev start -d`).
- 💥[Improvement] Make it possible to run `tutor k8s exec <command with multiple arguments>` (#636). As a consequence, it is no longer possible to run quoted commands: `tutor k8s exec "<some command>"`. Instead, you should remove the quotes: `tutor k8s exec <some command>`.
- 💥[Deprecation] Drop support for the `TUTOR_EDX_PLATFORM_SETTINGS` environment variable. It is now recommended to create a plugin instead.

66
docs/dev.rst
View File

@ -107,9 +107,43 @@ Sharing directories with containers
It may sometimes be convenient to mount container directories on the host, for instance: for editing and debugging. Tutor provides different solutions to this problem.
.. _mount_option:
Bind-mount volumes with ``--mount``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The `quickstart`, ``run``, ``runserver``, ``init`` and ``start`` subcommands of ``tutor dev`` and ``tutor local`` support the ``-m/--mount`` option (see :option:`tutor dev start -m`) which can take two different forms. The first is explicit::
tutor dev start --mount=lms:/path/to/edx-platform:/openedx/edx-platform lms
And the second is implicit::
tutor dev start --mount=/path/to/edx-platform lms
With the explicit form, the ``--mount`` option means "bind-mount the host folder /path/to/edx-platform to /openedx/edx-platform in the lms container".
If you use the explicit format, you will quickly realise that you usually want to bind-mount folders in multiple containers at a time. For instance, you will want to bind-mount the edx-platform repository in the "cms" container. To do that, write instead::
tutor dev start --mount=lms,cms:/path/to/edx-platform:/openedx/edx-platform lms
This command line can become cumbersome and inconvenient to work with. But Tutor can be smart about bind-mounting folders to the right containers in the right place when you use the implicit form of the ``--mount`` option. For instance, the following commands are equivalent::
# Explicit form
tutor dev start --mount=lms,lms-worker,lms-job,cms,cms-worker,cms-job:/path/to/edx-platform:/openedx/edx-platform lms
# Implicit form
tutor dev start --mount=/path/to/edx-platform lms
So, when should you *not* be using the implicit form? That would be when Tutor does not know where to bind-mount your host folders. For instance, if you wanted to bind-mount your edx-platform virtual environment located in ``~/venvs/edx-platform``, you should not write ``--mount=~/venvs/edx-platform``, because that folder would be mounted in a way that would override the edx-platform repository in the container. Instead, you should write::
tutor dev start --mount=lms:~/venvs/edx-platform:/openedx/venv lms
.. note:: Remember to setup your edx-platform repository for development! See :ref:`edx_platform_dev_env`.
Bind-mount from the "volumes/" directory
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. warning:: Bind-mounting volumes with the ``bindmount`` command is no longer the default, recommended way of bind-mounting volumes from the host. Instead, see the :ref:`mount option <mount_option>`.
Tutor makes it easy to create a bind-mount from an existing container. First, copy the contents of a container directory with the ``bindmount`` command. For instance, to copy the virtual environment of the "lms" container::
tutor dev bindmount lms /openedx/venv
@ -128,6 +162,8 @@ Notice how the ``--volume=/openedx/venv`` option differs from `Docker syntax <ht
Manual bind-mount to any directory
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. warning:: Manually bind-mounting volumes with the ``--volume`` option makes it difficult to simultaneously bind-mount to multiple containers. Also, the ``--volume`` options are not compatible with ``start`` commands. For an alternative, see the :ref:`mount option <mount_option>`.
The above solution may not work for you if you already have an existing directory, outside of the "volumes/" directory, which you would like mounted in one of your containers. For instance, you may want to mount your copy of the `edx-platform <https://github.com/openedx/edx-platform/>`__ repository. In such cases, you can simply use the ``-v/--volume`` `Docker option <https://docs.docker.com/storage/volumes/#choose-the--v-or---mount-flag>`__::
tutor dev run --volume=/path/to/edx-platform:/openedx/edx-platform lms bash
@ -135,7 +171,7 @@ The above solution may not work for you if you already have an existing director
Override docker-compose volumes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The above solutions require that you explicitly pass the ``-v/--volume`` to every ``run`` or ``runserver`` command, which may be inconvenient. Also, these solutions are not compatible with the ``start`` command. To address these issues, you can create a ``docker-compose.override.yml`` file that will specify custom volumes to be used with all ``dev`` commands::
The above solutions require that you explicitly pass the ``-m/--mount`` options to every ``run``, ``runserver``, ``start`` or ``init`` command, which may be inconvenient. To address these issues, you can create a ``docker-compose.override.yml`` file that will specify custom volumes to be used with all ``dev`` commands::
vim "$(tutor config printroot)/env/dev/docker-compose.override.yml"
@ -156,37 +192,31 @@ You are then free to bind-mount any directory to any container. For instance, to
volumes:
- /path/to/edx-platform:/openedx/edx-platform
This override file will be loaded when running any ``tutor dev ..`` command. The edx-platform repo mounted at the specified path will be automatically mounted inside all LMS and CMS containers. With this file, you should no longer specify the ``-v/--volume`` option from the command line with the ``run`` or ``runserver`` commands.
This override file will be loaded when running any ``tutor dev ..`` command. The edx-platform repo mounted at the specified path will be automatically mounted inside all LMS and CMS containers. With this file, you should no longer specify the ``-m/--mount`` option from the command line.
.. note::
The ``tutor local`` commands load the ``docker-compose.override.yml`` file from the ``$(tutor config printroot)/env/local/docker-compose.override.yml`` directory.
The ``tutor local`` commands load the ``docker-compose.override.yml`` file from the ``$(tutor config printroot)/env/local/docker-compose.override.yml`` directory. One-time jobs from initialisation commands load the ``local/docker-compose.jobs.override.yml`` and ``dev/docker-compose.jobs.override.yml``.
Common tasks
------------
.. _edx_platform_dev_env:
Setting up a development environment for edx-platform
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Following the instructions :ref:`above <bind_mounts>` on how to bind-mount directories from the host above, you may mount your own `edx-platform <https://github.com/openedx/edx-platform/>`__ fork in your containers by running either::
Following the instructions :ref:`above <bind_mounts>` on how to bind-mount directories from the host above, you may mount your own `edx-platform <https://github.com/openedx/edx-platform/>`__ fork in your containers by running::
# Mount from the volumes/ directory
tutor dev bindmount lms /openedx/edx-platform
tutor dev runserver --volume=/openedx/edx-platform lms
tutor dev start -d --mount=/path/to/edx-platform lms
# Mount from an arbitrary directory
tutor dev runserver --volume=/path/to/edx-platform:/openedx/edx-platform lms
But to achieve that, you will have to make sure that your fork works with Tutor.
# Add your own volumes to $(tutor config printroot)/env/dev/docker-compose.override.yml
tutor dev runserver lms
If you choose any but the first solution above, you will have to make sure that your fork works with Tutor.
First of all, you should make sure that you are working off the ``open-release/maple.2`` tag. See the :ref:`fork edx-platform section <edx_platform_fork>` for more information.
First of all, you should make sure that you are working off the latest release tag (unless you are running the Tutor :ref:`nightly <nightly>` branch). See the :ref:`fork edx-platform section <edx_platform_fork>` for more information.
Then, you should run the following commands::
# Run bash in the lms container
tutor dev run [--volume=...] lms bash
tutor dev run --mount=/path/to/edx-platform lms bash
# Compile local python requirements
pip install --requirement requirements/edx/development.txt
@ -197,9 +227,9 @@ Then, you should run the following commands::
# Rebuild static assets
openedx-assets build --env=dev
To debug a local edx-platform repository, add a ``import ipdb; ipdb.set_trace()`` breakpoint anywhere in your code and run::
After running all these commands, your edx-platform repository will be ready for local development. To debug a local edx-platform repository, you can then add a ``import ipdb; ipdb.set_trace()`` breakpoint anywhere in your code and run::
tutor dev runserver [--volume=...] lms
tutor dev runserver --mount=/path/to/edx-platform lms
XBlock and edx-platform plugin development
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

30
tests/commands/test_compose.py Normal file
View File

@ -0,0 +1,30 @@
import unittest
from click.exceptions import ClickException
from tutor.commands import compose
class ComposeTests(unittest.TestCase):
def test_mount_option_parsing(self) -> None:
param = compose.MountParam()
self.assertEqual(
[("lms", "/path/to/edx-platform", "/openedx/edx-platform")],
param("lms:/path/to/edx-platform:/openedx/edx-platform"),
)
self.assertEqual(
[
("lms", "/path/to/edx-platform", "/openedx/edx-platform"),
("cms", "/path/to/edx-platform", "/openedx/edx-platform"),
],
param("lms,cms:/path/to/edx-platform:/openedx/edx-platform"),
)
self.assertEqual(
[
("lms", "/path/to/edx-platform", "/openedx/edx-platform"),
("cms", "/path/to/edx-platform", "/openedx/edx-platform"),
],
param("lms, cms:/path/to/edx-platform:/openedx/edx-platform"),
)
with self.assertRaises(ClickException):
param("lms,:/path/to/edx-platform:/openedx/edx-platform")

2
tutor/bindmounts.py
View File

@ -36,7 +36,7 @@ chown -R {user_id} {volumes_path}/{volume_name}""".format(
"--no-deps",
"--user=0",
"--volume",
"{}:{}".format(volumes_root_path, container_volumes_root_path),
f"{volumes_root_path}:{container_volumes_root_path}",
service,
"sh",
"-e",

206
tutor/commands/compose.py
View File

@ -1,5 +1,6 @@
import os
from typing import List
import re
import typing as t
import click
@ -7,22 +8,25 @@ from tutor import bindmounts
from tutor import config as tutor_config
from tutor import env as tutor_env
from tutor import fmt, jobs, utils
from tutor.commands.context import BaseJobContext
from tutor import serialize
from tutor.exceptions import TutorError
from tutor.types import Config
from tutor.commands.context import BaseJobContext
from tutor import hooks
class ComposeJobRunner(jobs.BaseComposeJobRunner):
def __init__(self, root: str, config: Config):
super().__init__(root, config)
self.project_name = ""
self.docker_compose_files: List[str] = []
self.docker_compose_job_files: List[str] = []
self.docker_compose_files: t.List[str] = []
self.docker_compose_job_files: t.List[str] = []
def docker_compose(self, *command: str) -> int:
"""
Run docker-compose with the right yml files.
"""
self.__update_docker_compose_tmp()
args = []
for docker_compose_path in self.docker_compose_files:
if os.path.exists(docker_compose_path):
@ -31,6 +35,35 @@ class ComposeJobRunner(jobs.BaseComposeJobRunner):
*args, "--project-name", self.project_name, *command
)
def __update_docker_compose_tmp(self) -> None:
"""
Update the contents of the docker-compose.tmp.yml file, which is generated at runtime.
"""
docker_compose_tmp = {
"version": "{{ DOCKER_COMPOSE_VERSION }}",
"services": {},
}
docker_compose_jobs_tmp = {
"version": "{{ DOCKER_COMPOSE_VERSION }}",
"services": {},
}
docker_compose_tmp = hooks.Filters.COMPOSE_LOCAL_TMP.apply(docker_compose_tmp)
docker_compose_jobs_tmp = hooks.Filters.COMPOSE_LOCAL_JOBS_TMP.apply(
docker_compose_jobs_tmp
)
docker_compose_tmp = tutor_env.render_unknown(self.config, docker_compose_tmp)
docker_compose_jobs_tmp = tutor_env.render_unknown(
self.config, docker_compose_jobs_tmp
)
tutor_env.write_to(
serialize.dumps(docker_compose_tmp),
tutor_env.pathjoin(self.root, "local", "docker-compose.tmp.yml"),
)
tutor_env.write_to(
serialize.dumps(docker_compose_jobs_tmp),
tutor_env.pathjoin(self.root, "local", "docker-compose.jobs.tmp.yml"),
)
def run_job(self, service: str, command: str) -> int:
"""
Run the "{{ service }}-job" service from local/docker-compose.jobs.yml with the
@ -60,16 +93,78 @@ class BaseComposeContext(BaseJobContext):
raise NotImplementedError
class MountParam(click.ParamType):
"""
Parser for --mount arguments of the form "service1[,service2,...]:/host/path:/container/path".
"""
name = "mount"
MountType = t.Tuple[str, str, str]
# Note that this syntax does not allow us to include colon ':' characters in paths
PARAM_REGEXP = (
r"(?P<services>[a-zA-Z0-9_, ]+):(?P<host_path>[^:]+):(?P<container_path>[^:]+)"
)
def convert(
self,
value: str,
param: t.Optional["click.Parameter"],
ctx: t.Optional[click.Context],
) -> t.List["MountType"]:
mounts: t.List["MountParam.MountType"] = []
match = re.match(self.PARAM_REGEXP, value)
if match:
# Argument is of the form "containers:/host/path:/container/path"
services: t.List[str] = [
service.strip() for service in match["services"].split(",")
]
host_path = os.path.abspath(os.path.expanduser(match["host_path"]))
host_path = host_path.replace(os.path.sep, "/")
container_path = match["container_path"]
for service in services:
if not service:
self.fail(
f"incorrect services syntax: '{match['services']}'", param, ctx
)
mounts.append((service, host_path, container_path))
else:
# Argument is of the form "/host/path"
host_path = os.path.abspath(os.path.expanduser(value))
volumes: t.Iterator[
t.Tuple[str, str]
] = hooks.Filters.COMPOSE_MOUNTS.iterate(os.path.basename(host_path))
for service, container_path in volumes:
mounts.append((service, host_path, container_path))
if not mounts:
raise self.fail(f"no mount found for {value}", param, ctx)
return mounts
mount_option = click.option(
"-m",
"--mount",
"mounts",
help="""Bind-mount a folder from the host in the right containers. This option can take two different forms. The first one is explicit: 'service1[,service2...]:/host/path:/container/path'. The other is implicit: '/host/path'. Arguments passed in the implicit form will be parsed by plugins to define the right folders to bind-mount from the host.""",
type=MountParam(),
multiple=True,
)
@click.command(
short_help="Run all or a selection of services.",
help="Run all or a selection of services. Docker images will be rebuilt where necessary.",
)
@click.option("--skip-build", is_flag=True, help="Skip image building")
@click.option("-d", "--detach", is_flag=True, help="Start in daemon mode")
@mount_option
@click.argument("services", metavar="service", nargs=-1)
@click.pass_obj
def start(
context: BaseComposeContext, skip_build: bool, detach: bool, services: List[str]
context: BaseComposeContext,
skip_build: bool,
detach: bool,
mounts: t.Tuple[t.List[MountParam.MountType]],
services: t.List[str],
) -> None:
command = ["up", "--remove-orphans"]
if not skip_build:
@ -77,6 +172,8 @@ def start(
if detach:
command.append("-d")
process_mount_arguments(mounts)
# Start services
config = tutor_config.load(context.root)
context.job_runner(config).docker_compose(*command, *services)
@ -85,7 +182,7 @@ def start(
@click.command(help="Stop a running platform")
@click.argument("services", metavar="service", nargs=-1)
@click.pass_obj
def stop(context: BaseComposeContext, services: List[str]) -> None:
def stop(context: BaseComposeContext, services: t.List[str]) -> None:
config = tutor_config.load(context.root)
context.job_runner(config).docker_compose("stop", *services)
@ -97,7 +194,7 @@ def stop(context: BaseComposeContext, services: List[str]) -> None:
@click.option("-d", "--detach", is_flag=True, help="Start in daemon mode")
@click.argument("services", metavar="service", nargs=-1)
@click.pass_context
def reboot(context: click.Context, detach: bool, services: List[str]) -> None:
def reboot(context: click.Context, detach: bool, services: t.List[str]) -> None:
context.invoke(stop, services=services)
context.invoke(start, detach=detach, services=services)
@ -111,7 +208,7 @@ fully stop the platform, use the 'reboot' command.""",
)
@click.argument("services", metavar="service", nargs=-1)
@click.pass_obj
def restart(context: BaseComposeContext, services: List[str]) -> None:
def restart(context: BaseComposeContext, services: t.List[str]) -> None:
config = tutor_config.load(context.root)
command = ["restart"]
if "all" in services:
@ -130,8 +227,14 @@ def restart(context: BaseComposeContext, services: List[str]) -> None:
@click.command(help="Initialise all applications")
@click.option("-l", "--limit", help="Limit initialisation to this service or plugin")
@mount_option
@click.pass_obj
def init(context: BaseComposeContext, limit: str) -> None:
def init(
context: BaseComposeContext,
limit: str,
mounts: t.Tuple[t.List[MountParam.MountType]],
) -> None:
process_mount_arguments(mounts)
config = tutor_config.load(context.root)
runner = context.job_runner(config)
jobs.initialise(runner, limit_to=limit)
@ -177,7 +280,9 @@ def createuser(
)
@click.argument("theme_name")
@click.pass_obj
def settheme(context: BaseComposeContext, domains: List[str], theme_name: str) -> None:
def settheme(
context: BaseComposeContext, domains: t.List[str], theme_name: str
) -> None:
config = tutor_config.load(context.root)
runner = context.job_runner(config)
domains = domains or jobs.get_all_openedx_domains(config)
@ -202,9 +307,15 @@ def importdemocourse(context: BaseComposeContext) -> None:
),
context_settings={"ignore_unknown_options": True},
)
@mount_option
@click.argument("args", nargs=-1, required=True)
@click.pass_context
def run(context: click.Context, args: List[str]) -> None:
def run(
context: click.Context,
mounts: t.Tuple[t.List[MountParam.MountType]],
args: t.List[str],
) -> None:
process_mount_arguments(mounts)
extra_args = ["--rm"]
if not utils.is_a_tty():
extra_args.append("-T")
@ -221,6 +332,9 @@ def run(context: click.Context, args: List[str]) -> None:
@click.argument("path")
@click.pass_obj
def bindmount_command(context: BaseComposeContext, service: str, path: str) -> None:
"""
This command is made obsolete by the --mount arguments.
"""
config = tutor_config.load(context.root)
host_path = bindmounts.create(context.job_runner(config), service, path)
fmt.echo_info(
@ -241,7 +355,7 @@ def bindmount_command(context: BaseComposeContext, service: str, path: str) -> N
)
@click.argument("args", nargs=-1, required=True)
@click.pass_context
def execute(context: click.Context, args: List[str]) -> None:
def execute(context: click.Context, args: t.List[str]) -> None:
context.invoke(dc_command, command="exec", args=args)
@ -281,7 +395,7 @@ def status(context: click.Context) -> None:
@click.argument("command")
@click.argument("args", nargs=-1)
@click.pass_obj
def dc_command(context: BaseComposeContext, command: str, args: List[str]) -> None:
def dc_command(context: BaseComposeContext, command: str, args: t.List[str]) -> None:
config = tutor_config.load(context.root)
volumes, non_volume_args = bindmounts.parse_volumes(args)
volume_args = []
@ -299,6 +413,72 @@ def dc_command(context: BaseComposeContext, command: str, args: List[str]) -> No
context.job_runner(config).docker_compose(command, *volume_args, *non_volume_args)
def process_mount_arguments(mounts: t.Tuple[t.List[MountParam.MountType]]) -> None:
"""
Process --mount arguments.
Most docker-compose commands support --mount arguments. This option
is used to bind-mount folders from the host. A docker-compose.tmp.yml is
generated at runtime and includes the bind-mounted volumes that were passed as CLI
arguments.
Bind-mounts that are associated to "*-job" services will be added to the
docker-compose jobs file.
"""
app_mounts: t.List[MountParam.MountType] = []
job_mounts: t.List[MountParam.MountType] = []
for mount in mounts:
for service, host_path, container_path in mount:
if service.endswith("-job"):
job_mounts.append((service, host_path, container_path))
else:
app_mounts.append((service, host_path, container_path))
def _add_mounts(
docker_compose: t.Dict[str, t.Any], bind_mounts: t.List[MountParam.MountType]
) -> t.Dict[str, t.Any]:
services = docker_compose.setdefault("services", {})
for service, host_path, container_path in bind_mounts:
fmt.echo_info(f"Bind-mount: {host_path} -> {container_path} in {service}")
services.setdefault(service, {"volumes": []})
services[service]["volumes"].append(f"{host_path}:{container_path}")
return docker_compose
# Save bind-mounts
@hooks.Filters.COMPOSE_LOCAL_TMP.add()
def _add_mounts_to_docker_compose_tmp(
docker_compose_tmp: t.Dict[str, t.Any]
) -> t.Dict[str, t.Any]:
return _add_mounts(docker_compose_tmp, app_mounts)
@hooks.Filters.COMPOSE_LOCAL_JOBS_TMP.add()
def _add_mounts_to_docker_compose_jobs_tmp(
docker_compose_tmp: t.Dict[str, t.Any]
) -> t.Dict[str, t.Any]:
return _add_mounts(docker_compose_tmp, job_mounts)
@hooks.Filters.COMPOSE_MOUNTS.add()
def _mount_edx_platform(
volumes: t.List[t.Tuple[str, str]], name: str
) -> t.List[t.Tuple[str, str]]:
"""
When mounting edx-platform with `--mount=/path/to/edx-platform`, bind-mount the host
repo in the lms/cms containers.
"""
if name == "edx-platform":
path = "/openedx/edx-platform"
volumes += [
("lms", path),
("cms", path),
("lms-worker", path),
("cms-worker", path),
("lms-job", path),
("cms-job", path),
]
return volumes
def add_commands(command_group: click.Group) -> None:
command_group.add_command(start)
command_group.add_command(stop)

18
tutor/commands/dev.py
View File

@ -1,4 +1,4 @@
from typing import List
import typing as t
import click
@ -21,12 +21,14 @@ class DevJobRunner(compose.ComposeJobRunner):
self.docker_compose_files += [
tutor_env.pathjoin(self.root, "local", "docker-compose.yml"),
tutor_env.pathjoin(self.root, "dev", "docker-compose.yml"),
tutor_env.pathjoin(self.root, "local", "docker-compose.tmp.yml"),
tutor_env.pathjoin(self.root, "local", "docker-compose.override.yml"),
tutor_env.pathjoin(self.root, "dev", "docker-compose.override.yml"),
]
self.docker_compose_job_files += [
tutor_env.pathjoin(self.root, "local", "docker-compose.jobs.yml"),
tutor_env.pathjoin(self.root, "dev", "docker-compose.jobs.yml"),
tutor_env.pathjoin(self.root, "local", "docker-compose.jobs.tmp.yml"),
tutor_env.pathjoin(self.root, "local", "docker-compose.jobs.override.yml"),
tutor_env.pathjoin(self.root, "dev", "docker-compose.jobs.override.yml"),
]
@ -99,21 +101,25 @@ Your Open edX platform is ready and can be accessed at the following urls:
help="Run a development server",
context_settings={"ignore_unknown_options": True},
)
@compose.mount_option
@click.argument("options", nargs=-1, required=False)
@click.argument("service")
@click.pass_context
def runserver(context: click.Context, options: List[str], service: str) -> None:
def runserver(
context: click.Context,
mounts: t.Tuple[t.List[compose.MountParam.MountType]],
options: t.List[str],
service: str,
) -> None:
config = tutor_config.load(context.obj.root)
if service in ["lms", "cms"]:
port = 8000 if service == "lms" else 8001
host = config["LMS_HOST"] if service == "lms" else config["CMS_HOST"]
fmt.echo_info(
"The {} service will be available at http://{}:{}".format(
service, host, port
)
f"The {service} service will be available at http://{host}:{port}"
)
args = ["--service-ports", *options, service]
context.invoke(compose.run, args=args)
context.invoke(compose.run, mounts=mounts, args=args)
dev.add_command(quickstart)

4
tutor/commands/images.py
View File

@ -174,7 +174,7 @@ def find_images_to_build(
] = hooks.Filters.IMAGES_BUILD.iterate(config)
found = False
for name, path, tag, args in all_images_to_build:
if name == image or image == "all":
if image in [name, "all"]:
found = True
tag = tutor_env.render_str(config, tag)
yield (name, path, tag, args)
@ -196,7 +196,7 @@ def find_remote_image_tags(
all_remote_images: t.Iterator[t.Tuple[str, str]] = filtre.iterate(config)
found = False
for name, tag in all_remote_images:
if name == image or image == "all":
if image in [name, "all"]:
found = True
yield tutor_env.render_str(config, tag)
if not found:

17
tutor/commands/local.py
View File

@ -1,4 +1,4 @@
from typing import Optional
import typing as t
import click
@ -22,10 +22,12 @@ class LocalJobRunner(compose.ComposeJobRunner):
self.docker_compose_files += [
tutor_env.pathjoin(self.root, "local", "docker-compose.yml"),
tutor_env.pathjoin(self.root, "local", "docker-compose.prod.yml"),
tutor_env.pathjoin(self.root, "local", "docker-compose.tmp.yml"),
tutor_env.pathjoin(self.root, "local", "docker-compose.override.yml"),
]
self.docker_compose_job_files += [
tutor_env.pathjoin(self.root, "local", "docker-compose.jobs.yml"),
tutor_env.pathjoin(self.root, "local", "docker-compose.jobs.tmp.yml"),
tutor_env.pathjoin(self.root, "local", "docker-compose.jobs.override.yml"),
]
@ -46,7 +48,12 @@ def local(context: click.Context) -> None:
@click.option("-I", "--non-interactive", is_flag=True, help="Run non-interactively")
@click.option("-p", "--pullimages", is_flag=True, help="Update docker images")
@click.pass_context
def quickstart(context: click.Context, non_interactive: bool, pullimages: bool) -> None:
def quickstart(
context: click.Context,
mounts: t.Tuple[t.List[compose.MountParam.MountType]],
non_interactive: bool,
pullimages: bool,
) -> None:
try:
utils.check_macos_docker_memory()
except exceptions.TutorError as e:
@ -113,9 +120,9 @@ Press enter when you are ready to continue"""
click.echo(fmt.title("Docker image updates"))
context.invoke(compose.dc_command, command="pull")
click.echo(fmt.title("Starting the platform in detached mode"))
context.invoke(compose.start, detach=True)
context.invoke(compose.start, mounts=mounts, detach=True)
click.echo(fmt.title("Database creation and migrations"))
context.invoke(compose.init)
context.invoke(compose.init, mounts=mounts)
config = tutor_config.load(context.obj.root)
fmt.echo_info(
@ -142,7 +149,7 @@ Your Open edX platform is ready and can be accessed at the following urls:
type=click.Choice(["ironwood", "juniper", "koa", "lilac"]),
)
@click.pass_context
def upgrade(context: click.Context, from_release: Optional[str]) -> None:
def upgrade(context: click.Context, from_release: t.Optional[str]) -> None:
fmt.echo_alert(
"This command only performs a partial upgrade of your Open edX platform. "
"To perform a full upgrade, you should run `tutor local quickstart`."

66
tutor/hooks/consts.py
View File

@ -97,6 +97,55 @@ class Filters:
return items
"""
#: List of commands to be executed during initialization. These commands typically
#: include database migrations, setting feature flags, etc.
#:
#: :parameter list[tuple[str, tuple[str, ...]]] tasks: list of ``(service, path)`` tasks.
#:
#: - ``service`` is the name of the container in which the task will be executed.
#: - ``path`` is a tuple that corresponds to a template relative path.
#: Example: ``("myplugin", "hooks", "myservice", "pre-init")`` (see:py:data:`IMAGES_BUILD`).
#: The command to execute will be read from that template, after it is rendered.
COMMANDS_INIT = filters.get("commands:init")
#: List of commands to be executed prior to initialization. These commands are run even
#: before the mysql databases are created and the migrations are applied.
#:
#: :parameter list[tuple[str, tuple[str, ...]]] tasks: list of ``(service, path)`` tasks. (see :py:data:`COMMANDS_INIT`).
COMMANDS_PRE_INIT = filters.get("commands:pre-init")
#: List of folders to bind-mount in docker-compose containers, either in ``tutor local`` or ``tutor dev``.
#:
#: Many ``tutor local`` and ``tutor dev`` commands support ``--mounts`` options
#: that allow plugins to define custom behaviour at runtime. For instance
#: ``--mount=/path/to/edx-platform`` would cause this host folder to be
#: bind-mounted in different containers (lms, lms-worker, cms, cms-worker) at the
#: /openedx/edx-platform location. Plugin developers may implement this filter to
#: define custom behaviour when mounting folders that relate to their plugins. For
#: instance, the ecommerce plugin may process the ``--mount=/path/to/ecommerce``
#: option.
#:
#: :parameter list[tuple[str, str]] mounts: each item is a ``(service, path)``
#: tuple, where ``service`` is the name of the docker-compose service and ``path`` is
#: the location in the container where the folder should be bind-mounted. Note: the
#: path must be slash-separated ("/"). Thus, do not use ``os.path.join`` to generate
#: the ``path`` because it will fail on Windows.
#: :parameter str name: basename of the host-mounted folder. In the example above,
#: this is "edx-platform". When implementing this filter you should check this name to
#: conditionnally add mounts.
COMPOSE_MOUNTS = filters.get("compose:mounts")
#: Contents of the local/docker-compose.tmp.yml file that will be generated at
#: runtime. This is used for instance to bind-mount folders from the host (see
#: :py:data:`COMPOSE_MOUNTS`)
#:
#: :parameter dict[str, ...] docker_compose_tmp: values which will be serialized to local/docker-compose.tmp.yml.
#: Keys and values will be rendered before saving, such that you may include ``{{ ... }}`` statements.
COMPOSE_LOCAL_TMP = filters.get("compose:local:tmp")
#: Same as :py:data:`COMPOSE_LOCAL_TMP` but for jobs
COMPOSE_LOCAL_JOBS_TMP = filters.get("compose:local-jobs:tmp")
#: List of images to be built when we run ``tutor images build ...``.
#:
#: :parameter list[tuple[str, tuple[str, ...], str, tuple[str, ...]]] tasks: list of ``(name, path, tag, args)`` tuples.
@ -125,23 +174,6 @@ class Filters:
#: Parameters are the same as for :py:data:`IMAGES_PULL`.
IMAGES_PUSH = filters.get("images:push")
#: List of commands to be executed during initialization. These commands typically
#: include database migrations, setting feature flags, etc.
#:
#: :parameter list[tuple[str, tuple[str, ...]]] tasks: list of ``(service, path)`` tasks.
#:
#: - ``service`` is the name of the container in which the task will be executed.
#: - ``path`` is a tuple that corresponds to a template relative path.
#: Example: ``("myplugin", "hooks", "myservice", "pre-init")`` (see:py:data:`IMAGES_BUILD`).
#: The command to execute will be read from that template, after it is rendered.
COMMANDS_INIT = filters.get("commands:init")
#: List of commands to be executed prior to initialization. These commands are run even
#: before the mysql databases are created and the migrations are applied.
#:
#: :parameter list[tuple[str, tuple[str, ...]]] tasks: list of ``(service, path)`` tasks. (see :py:data:`COMMANDS_INIT`).
COMMANDS_PRE_INIT = filters.get("commands:pre-init")
#: List of command line interface (CLI) commands.
#:
#: :parameter list commands: commands are instances of ``click.Command``. They will

1
tutor/templates/config/defaults.yml
View File

@ -8,6 +8,7 @@ CMS_OAUTH2_KEY_SSO: "cms-sso"
CMS_OAUTH2_KEY_SSO_DEV: "cms-sso-dev"
CONTACT_EMAIL: "contact@{{ LMS_HOST }}"
DEV_PROJECT_NAME: "{{ TUTOR_APP }}_dev"
DOCKER_COMPOSE_VERSION: "3.7"
DOCKER_REGISTRY: "docker.io/"
DOCKER_IMAGE_OPENEDX: "{{ DOCKER_REGISTRY }}overhangio/openedx:{{ TUTOR_VERSION }}"
DOCKER_IMAGE_OPENEDX_DEV: "openedx-dev"

2
tutor/templates/dev/docker-compose.jobs.yml
View File

@ -1,3 +1,3 @@
version: "3.7"
version: "{{ DOCKER_COMPOSE_VERSION }}"
services: {% if not patch("dev-docker-compose-jobs-services") %}{}{% endif %}
{{ patch("dev-docker-compose-jobs-services")|indent(4) }}

2
tutor/templates/dev/docker-compose.yml
View File

@ -1,4 +1,4 @@
version: "3.7"
version: "{{ DOCKER_COMPOSE_VERSION }}"
x-openedx-service:
&openedx-service

2
tutor/templates/local/docker-compose.jobs.yml
View File

@ -1,4 +1,4 @@
version: "3.7"
version: "{{ DOCKER_COMPOSE_VERSION }}"
services:
mysql-job:

2
tutor/templates/local/docker-compose.prod.yml
View File

@ -1,4 +1,4 @@
version: "3.7"
version: "{{ DOCKER_COMPOSE_VERSION }}"
services:
# Web proxy for load balancing and SSL termination
caddy:

2
tutor/templates/local/docker-compose.yml
View File

@ -1,4 +1,4 @@
version: "3.7"
version: "{{ DOCKER_COMPOSE_VERSION }}"
services:
############# External services