mirror of
https://github.com/ChristianLight/tutor.git
synced 2024-12-12 14:17:46 +00:00
Merge branch 'master' into nightly
This commit is contained in:
Régis Behmo
commit
224cd6c19f
@ -4,6 +4,8 @@ Note: Breaking changes between versions are indicated by "💥".
|
||||
|
||||
## Unreleased
|
||||
|
||||
- [Security] Apply rate limiting security fix (see [commit](https://github.com/overhangio/edx-platform/commit/b5723e416e628cac4fa84392ca13e1b72817674f)).
|
||||
- [Feature] Introduce the ``-m/--mount`` option in ``local`` and ``dev`` commands to auto-magically bind-mount folders from the host.
|
||||
- [Feature] Add `tutor dev quickstart` command, which is similar to `tutor local quickstart`, except that it uses dev containers instead of local production ones and includes some other small differences for the convience of Open edX developers. This should remove some friction from the Open edX development setup process, which previously required that users provision using local producation containers (`tutor local quickstart`) but then stop them and switch to dev containers (`tutor local stop && tutor dev start -d`).
|
||||
- 💥[Improvement] Make it possible to run `tutor k8s exec <command with multiple arguments>` (#636). As a consequence, it is no longer possible to run quoted commands: `tutor k8s exec "<some command>"`. Instead, you should remove the quotes: `tutor k8s exec <some command>`.
|
||||
- 💥[Deprecation] Drop support for the `TUTOR_EDX_PLATFORM_SETTINGS` environment variable. It is now recommended to create a plugin instead.
|
||||
|
||||
66
docs/dev.rst
66
docs/dev.rst
@ -107,9 +107,43 @@ Sharing directories with containers
|
||||
|
||||
It may sometimes be convenient to mount container directories on the host, for instance: for editing and debugging. Tutor provides different solutions to this problem.
|
||||
|
||||
.. _mount_option:
|
||||
|
||||
Bind-mount volumes with ``--mount``
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The `quickstart`, ``run``, ``runserver``, ``init`` and ``start`` subcommands of ``tutor dev`` and ``tutor local`` support the ``-m/--mount`` option (see :option:`tutor dev start -m`) which can take two different forms. The first is explicit::
|
||||
|
||||
tutor dev start --mount=lms:/path/to/edx-platform:/openedx/edx-platform lms
|
||||
|
||||
And the second is implicit::
|
||||
|
||||
tutor dev start --mount=/path/to/edx-platform lms
|
||||
|
||||
With the explicit form, the ``--mount`` option means "bind-mount the host folder /path/to/edx-platform to /openedx/edx-platform in the lms container".
|
||||
|
||||
If you use the explicit format, you will quickly realise that you usually want to bind-mount folders in multiple containers at a time. For instance, you will want to bind-mount the edx-platform repository in the "cms" container. To do that, write instead::
|
||||
|
||||
tutor dev start --mount=lms,cms:/path/to/edx-platform:/openedx/edx-platform lms
|
||||
|
||||
This command line can become cumbersome and inconvenient to work with. But Tutor can be smart about bind-mounting folders to the right containers in the right place when you use the implicit form of the ``--mount`` option. For instance, the following commands are equivalent::
|
||||
|
||||
# Explicit form
|
||||
tutor dev start --mount=lms,lms-worker,lms-job,cms,cms-worker,cms-job:/path/to/edx-platform:/openedx/edx-platform lms
|
||||
# Implicit form
|
||||
tutor dev start --mount=/path/to/edx-platform lms
|
||||
|
||||
So, when should you *not* be using the implicit form? That would be when Tutor does not know where to bind-mount your host folders. For instance, if you wanted to bind-mount your edx-platform virtual environment located in ``~/venvs/edx-platform``, you should not write ``--mount=~/venvs/edx-platform``, because that folder would be mounted in a way that would override the edx-platform repository in the container. Instead, you should write::
|
||||
|
||||
tutor dev start --mount=lms:~/venvs/edx-platform:/openedx/venv lms
|
||||
|
||||
.. note:: Remember to setup your edx-platform repository for development! See :ref:`edx_platform_dev_env`.
|
||||
|
||||
Bind-mount from the "volumes/" directory
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. warning:: Bind-mounting volumes with the ``bindmount`` command is no longer the default, recommended way of bind-mounting volumes from the host. Instead, see the :ref:`mount option <mount_option>`.
|
||||
|
||||
Tutor makes it easy to create a bind-mount from an existing container. First, copy the contents of a container directory with the ``bindmount`` command. For instance, to copy the virtual environment of the "lms" container::
|
||||
|
||||
tutor dev bindmount lms /openedx/venv
|
||||
@ -128,6 +162,8 @@ Notice how the ``--volume=/openedx/venv`` option differs from `Docker syntax <ht
|
||||
Manual bind-mount to any directory
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. warning:: Manually bind-mounting volumes with the ``--volume`` option makes it difficult to simultaneously bind-mount to multiple containers. Also, the ``--volume`` options are not compatible with ``start`` commands. For an alternative, see the :ref:`mount option <mount_option>`.
|
||||
|
||||
The above solution may not work for you if you already have an existing directory, outside of the "volumes/" directory, which you would like mounted in one of your containers. For instance, you may want to mount your copy of the `edx-platform <https://github.com/openedx/edx-platform/>`__ repository. In such cases, you can simply use the ``-v/--volume`` `Docker option <https://docs.docker.com/storage/volumes/#choose-the--v-or---mount-flag>`__::
|
||||
|
||||
tutor dev run --volume=/path/to/edx-platform:/openedx/edx-platform lms bash
|
||||
@ -135,7 +171,7 @@ The above solution may not work for you if you already have an existing director
|
||||
Override docker-compose volumes
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The above solutions require that you explicitly pass the ``-v/--volume`` to every ``run`` or ``runserver`` command, which may be inconvenient. Also, these solutions are not compatible with the ``start`` command. To address these issues, you can create a ``docker-compose.override.yml`` file that will specify custom volumes to be used with all ``dev`` commands::
|
||||
The above solutions require that you explicitly pass the ``-m/--mount`` options to every ``run``, ``runserver``, ``start`` or ``init`` command, which may be inconvenient. To address these issues, you can create a ``docker-compose.override.yml`` file that will specify custom volumes to be used with all ``dev`` commands::
|
||||
|
||||
vim "$(tutor config printroot)/env/dev/docker-compose.override.yml"
|
||||
|
||||
@ -156,37 +192,31 @@ You are then free to bind-mount any directory to any container. For instance, to
|
||||
volumes:
|
||||
- /path/to/edx-platform:/openedx/edx-platform
|
||||
|
||||
This override file will be loaded when running any ``tutor dev ..`` command. The edx-platform repo mounted at the specified path will be automatically mounted inside all LMS and CMS containers. With this file, you should no longer specify the ``-v/--volume`` option from the command line with the ``run`` or ``runserver`` commands.
|
||||
This override file will be loaded when running any ``tutor dev ..`` command. The edx-platform repo mounted at the specified path will be automatically mounted inside all LMS and CMS containers. With this file, you should no longer specify the ``-m/--mount`` option from the command line.
|
||||
|
||||
.. note::
|
||||
The ``tutor local`` commands load the ``docker-compose.override.yml`` file from the ``$(tutor config printroot)/env/local/docker-compose.override.yml`` directory.
|
||||
The ``tutor local`` commands load the ``docker-compose.override.yml`` file from the ``$(tutor config printroot)/env/local/docker-compose.override.yml`` directory. One-time jobs from initialisation commands load the ``local/docker-compose.jobs.override.yml`` and ``dev/docker-compose.jobs.override.yml``.
|
||||
|
||||
Common tasks
|
||||
------------
|
||||
|
||||
.. _edx_platform_dev_env:
|
||||
|
||||
Setting up a development environment for edx-platform
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Following the instructions :ref:`above <bind_mounts>` on how to bind-mount directories from the host above, you may mount your own `edx-platform <https://github.com/openedx/edx-platform/>`__ fork in your containers by running either::
|
||||
Following the instructions :ref:`above <bind_mounts>` on how to bind-mount directories from the host above, you may mount your own `edx-platform <https://github.com/openedx/edx-platform/>`__ fork in your containers by running::
|
||||
|
||||
# Mount from the volumes/ directory
|
||||
tutor dev bindmount lms /openedx/edx-platform
|
||||
tutor dev runserver --volume=/openedx/edx-platform lms
|
||||
tutor dev start -d --mount=/path/to/edx-platform lms
|
||||
|
||||
# Mount from an arbitrary directory
|
||||
tutor dev runserver --volume=/path/to/edx-platform:/openedx/edx-platform lms
|
||||
But to achieve that, you will have to make sure that your fork works with Tutor.
|
||||
|
||||
# Add your own volumes to $(tutor config printroot)/env/dev/docker-compose.override.yml
|
||||
tutor dev runserver lms
|
||||
|
||||
If you choose any but the first solution above, you will have to make sure that your fork works with Tutor.
|
||||
|
||||
First of all, you should make sure that you are working off the ``open-release/maple.2`` tag. See the :ref:`fork edx-platform section <edx_platform_fork>` for more information.
|
||||
First of all, you should make sure that you are working off the latest release tag (unless you are running the Tutor :ref:`nightly <nightly>` branch). See the :ref:`fork edx-platform section <edx_platform_fork>` for more information.
|
||||
|
||||
Then, you should run the following commands::
|
||||
|
||||
# Run bash in the lms container
|
||||
tutor dev run [--volume=...] lms bash
|
||||
tutor dev run --mount=/path/to/edx-platform lms bash
|
||||
|
||||
# Compile local python requirements
|
||||
pip install --requirement requirements/edx/development.txt
|
||||
@ -197,9 +227,9 @@ Then, you should run the following commands::
|
||||
# Rebuild static assets
|
||||
openedx-assets build --env=dev
|
||||
|
||||
To debug a local edx-platform repository, add a ``import ipdb; ipdb.set_trace()`` breakpoint anywhere in your code and run::
|
||||
After running all these commands, your edx-platform repository will be ready for local development. To debug a local edx-platform repository, you can then add a ``import ipdb; ipdb.set_trace()`` breakpoint anywhere in your code and run::
|
||||
|
||||
tutor dev runserver [--volume=...] lms
|
||||
tutor dev runserver --mount=/path/to/edx-platform lms
|
||||
|
||||
XBlock and edx-platform plugin development
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
30
tests/commands/test_compose.py
Normal file
30
tests/commands/test_compose.py
Normal file
@ -0,0 +1,30 @@
|
||||
import unittest
|
||||
|
||||
from click.exceptions import ClickException
|
||||
from tutor.commands import compose
|
||||
|
||||
|
||||
class ComposeTests(unittest.TestCase):
|
||||
def test_mount_option_parsing(self) -> None:
|
||||
param = compose.MountParam()
|
||||
|
||||
self.assertEqual(
|
||||
[("lms", "/path/to/edx-platform", "/openedx/edx-platform")],
|
||||
param("lms:/path/to/edx-platform:/openedx/edx-platform"),
|
||||
)
|
||||
self.assertEqual(
|
||||
[
|
||||
("lms", "/path/to/edx-platform", "/openedx/edx-platform"),
|
||||
("cms", "/path/to/edx-platform", "/openedx/edx-platform"),
|
||||
],
|
||||
param("lms,cms:/path/to/edx-platform:/openedx/edx-platform"),
|
||||
)
|
||||
self.assertEqual(
|
||||
[
|
||||
("lms", "/path/to/edx-platform", "/openedx/edx-platform"),
|
||||
("cms", "/path/to/edx-platform", "/openedx/edx-platform"),
|
||||
],
|
||||
param("lms, cms:/path/to/edx-platform:/openedx/edx-platform"),
|
||||
)
|
||||
with self.assertRaises(ClickException):
|
||||
param("lms,:/path/to/edx-platform:/openedx/edx-platform")
|
||||
@ -16,16 +16,9 @@ class JobsTests(unittest.TestCase):
|
||||
config = tutor_config.load_full(root)
|
||||
runner = context.job_runner(config)
|
||||
jobs.initialise(runner)
|
||||
|
||||
output = mock_stdout.getvalue().strip()
|
||||
service = re.search(r"Service: (\w*)", output)
|
||||
commands = re.search(r"(-----)([\S\s]+)(-----)", output)
|
||||
assert service is not None
|
||||
assert commands is not None
|
||||
self.assertTrue(output.startswith("Initialising all services..."))
|
||||
self.assertTrue(output.endswith("All services initialised."))
|
||||
self.assertEqual(service.group(1), "mysql")
|
||||
self.assertTrue(commands.group(2))
|
||||
|
||||
def test_create_user_command_without_staff(self) -> None:
|
||||
command = jobs.create_user_command("superuser", False, "username", "email")
|
||||
|
||||
@ -36,7 +36,7 @@ chown -R {user_id} {volumes_path}/{volume_name}""".format(
|
||||
"--no-deps",
|
||||
"--user=0",
|
||||
"--volume",
|
||||
"{}:{}".format(volumes_root_path, container_volumes_root_path),
|
||||
f"{volumes_root_path}:{container_volumes_root_path}",
|
||||
service,
|
||||
"sh",
|
||||
"-e",
|
||||
|
||||
@ -14,12 +14,13 @@ from tutor.commands.k8s import k8s
|
||||
from tutor.commands.local import local
|
||||
from tutor.commands.plugins import plugins_command
|
||||
|
||||
# Everyone on board
|
||||
hooks.Actions.CORE_READY.do()
|
||||
|
||||
|
||||
def main() -> None:
|
||||
try:
|
||||
# Everyone on board
|
||||
# Note that this action should not be triggered in the module scope, because it
|
||||
# makes it difficult for tests to rollback changes.
|
||||
hooks.Actions.CORE_READY.do()
|
||||
cli() # pylint: disable=no-value-for-parameter
|
||||
except KeyboardInterrupt:
|
||||
pass
|
||||
|
||||
@ -1,5 +1,6 @@
|
||||
import os
|
||||
from typing import List
|
||||
import re
|
||||
import typing as t
|
||||
|
||||
import click
|
||||
|
||||
@ -7,22 +8,25 @@ from tutor import bindmounts
|
||||
from tutor import config as tutor_config
|
||||
from tutor import env as tutor_env
|
||||
from tutor import fmt, jobs, utils
|
||||
from tutor.commands.context import BaseJobContext
|
||||
from tutor import serialize
|
||||
from tutor.exceptions import TutorError
|
||||
from tutor.types import Config
|
||||
from tutor.commands.context import BaseJobContext
|
||||
from tutor import hooks
|
||||
|
||||
|
||||
class ComposeJobRunner(jobs.BaseComposeJobRunner):
|
||||
def __init__(self, root: str, config: Config):
|
||||
super().__init__(root, config)
|
||||
self.project_name = ""
|
||||
self.docker_compose_files: List[str] = []
|
||||
self.docker_compose_job_files: List[str] = []
|
||||
self.docker_compose_files: t.List[str] = []
|
||||
self.docker_compose_job_files: t.List[str] = []
|
||||
|
||||
def docker_compose(self, *command: str) -> int:
|
||||
"""
|
||||
Run docker-compose with the right yml files.
|
||||
"""
|
||||
self.__update_docker_compose_tmp()
|
||||
args = []
|
||||
for docker_compose_path in self.docker_compose_files:
|
||||
if os.path.exists(docker_compose_path):
|
||||
@ -31,6 +35,35 @@ class ComposeJobRunner(jobs.BaseComposeJobRunner):
|
||||
*args, "--project-name", self.project_name, *command
|
||||
)
|
||||
|
||||
def __update_docker_compose_tmp(self) -> None:
|
||||
"""
|
||||
Update the contents of the docker-compose.tmp.yml file, which is generated at runtime.
|
||||
"""
|
||||
docker_compose_tmp = {
|
||||
"version": "{{ DOCKER_COMPOSE_VERSION }}",
|
||||
"services": {},
|
||||
}
|
||||
docker_compose_jobs_tmp = {
|
||||
"version": "{{ DOCKER_COMPOSE_VERSION }}",
|
||||
"services": {},
|
||||
}
|
||||
docker_compose_tmp = hooks.Filters.COMPOSE_LOCAL_TMP.apply(docker_compose_tmp)
|
||||
docker_compose_jobs_tmp = hooks.Filters.COMPOSE_LOCAL_JOBS_TMP.apply(
|
||||
docker_compose_jobs_tmp
|
||||
)
|
||||
docker_compose_tmp = tutor_env.render_unknown(self.config, docker_compose_tmp)
|
||||
docker_compose_jobs_tmp = tutor_env.render_unknown(
|
||||
self.config, docker_compose_jobs_tmp
|
||||
)
|
||||
tutor_env.write_to(
|
||||
serialize.dumps(docker_compose_tmp),
|
||||
tutor_env.pathjoin(self.root, "local", "docker-compose.tmp.yml"),
|
||||
)
|
||||
tutor_env.write_to(
|
||||
serialize.dumps(docker_compose_jobs_tmp),
|
||||
tutor_env.pathjoin(self.root, "local", "docker-compose.jobs.tmp.yml"),
|
||||
)
|
||||
|
||||
def run_job(self, service: str, command: str) -> int:
|
||||
"""
|
||||
Run the "{{ service }}-job" service from local/docker-compose.jobs.yml with the
|
||||
@ -60,16 +93,78 @@ class BaseComposeContext(BaseJobContext):
|
||||
raise NotImplementedError
|
||||
|
||||
|
||||
class MountParam(click.ParamType):
|
||||
"""
|
||||
Parser for --mount arguments of the form "service1[,service2,...]:/host/path:/container/path".
|
||||
"""
|
||||
|
||||
name = "mount"
|
||||
MountType = t.Tuple[str, str, str]
|
||||
# Note that this syntax does not allow us to include colon ':' characters in paths
|
||||
PARAM_REGEXP = (
|
||||
r"(?P<services>[a-zA-Z0-9_, ]+):(?P<host_path>[^:]+):(?P<container_path>[^:]+)"
|
||||
)
|
||||
|
||||
def convert(
|
||||
self,
|
||||
value: str,
|
||||
param: t.Optional["click.Parameter"],
|
||||
ctx: t.Optional[click.Context],
|
||||
) -> t.List["MountType"]:
|
||||
mounts: t.List["MountParam.MountType"] = []
|
||||
match = re.match(self.PARAM_REGEXP, value)
|
||||
if match:
|
||||
# Argument is of the form "containers:/host/path:/container/path"
|
||||
services: t.List[str] = [
|
||||
service.strip() for service in match["services"].split(",")
|
||||
]
|
||||
host_path = os.path.abspath(os.path.expanduser(match["host_path"]))
|
||||
host_path = host_path.replace(os.path.sep, "/")
|
||||
container_path = match["container_path"]
|
||||
for service in services:
|
||||
if not service:
|
||||
self.fail(
|
||||
f"incorrect services syntax: '{match['services']}'", param, ctx
|
||||
)
|
||||
mounts.append((service, host_path, container_path))
|
||||
else:
|
||||
# Argument is of the form "/host/path"
|
||||
host_path = os.path.abspath(os.path.expanduser(value))
|
||||
volumes: t.Iterator[
|
||||
t.Tuple[str, str]
|
||||
] = hooks.Filters.COMPOSE_MOUNTS.iterate(os.path.basename(host_path))
|
||||
for service, container_path in volumes:
|
||||
mounts.append((service, host_path, container_path))
|
||||
if not mounts:
|
||||
raise self.fail(f"no mount found for {value}", param, ctx)
|
||||
return mounts
|
||||
|
||||
|
||||
mount_option = click.option(
|
||||
"-m",
|
||||
"--mount",
|
||||
"mounts",
|
||||
help="""Bind-mount a folder from the host in the right containers. This option can take two different forms. The first one is explicit: 'service1[,service2...]:/host/path:/container/path'. The other is implicit: '/host/path'. Arguments passed in the implicit form will be parsed by plugins to define the right folders to bind-mount from the host.""",
|
||||
type=MountParam(),
|
||||
multiple=True,
|
||||
)
|
||||
|
||||
|
||||
@click.command(
|
||||
short_help="Run all or a selection of services.",
|
||||
help="Run all or a selection of services. Docker images will be rebuilt where necessary.",
|
||||
)
|
||||
@click.option("--skip-build", is_flag=True, help="Skip image building")
|
||||
@click.option("-d", "--detach", is_flag=True, help="Start in daemon mode")
|
||||
@mount_option
|
||||
@click.argument("services", metavar="service", nargs=-1)
|
||||
@click.pass_obj
|
||||
def start(
|
||||
context: BaseComposeContext, skip_build: bool, detach: bool, services: List[str]
|
||||
context: BaseComposeContext,
|
||||
skip_build: bool,
|
||||
detach: bool,
|
||||
mounts: t.Tuple[t.List[MountParam.MountType]],
|
||||
services: t.List[str],
|
||||
) -> None:
|
||||
command = ["up", "--remove-orphans"]
|
||||
if not skip_build:
|
||||
@ -77,6 +172,8 @@ def start(
|
||||
if detach:
|
||||
command.append("-d")
|
||||
|
||||
process_mount_arguments(mounts)
|
||||
|
||||
# Start services
|
||||
config = tutor_config.load(context.root)
|
||||
context.job_runner(config).docker_compose(*command, *services)
|
||||
@ -85,7 +182,7 @@ def start(
|
||||
@click.command(help="Stop a running platform")
|
||||
@click.argument("services", metavar="service", nargs=-1)
|
||||
@click.pass_obj
|
||||
def stop(context: BaseComposeContext, services: List[str]) -> None:
|
||||
def stop(context: BaseComposeContext, services: t.List[str]) -> None:
|
||||
config = tutor_config.load(context.root)
|
||||
context.job_runner(config).docker_compose("stop", *services)
|
||||
|
||||
@ -97,7 +194,7 @@ def stop(context: BaseComposeContext, services: List[str]) -> None:
|
||||
@click.option("-d", "--detach", is_flag=True, help="Start in daemon mode")
|
||||
@click.argument("services", metavar="service", nargs=-1)
|
||||
@click.pass_context
|
||||
def reboot(context: click.Context, detach: bool, services: List[str]) -> None:
|
||||
def reboot(context: click.Context, detach: bool, services: t.List[str]) -> None:
|
||||
context.invoke(stop, services=services)
|
||||
context.invoke(start, detach=detach, services=services)
|
||||
|
||||
@ -111,7 +208,7 @@ fully stop the platform, use the 'reboot' command.""",
|
||||
)
|
||||
@click.argument("services", metavar="service", nargs=-1)
|
||||
@click.pass_obj
|
||||
def restart(context: BaseComposeContext, services: List[str]) -> None:
|
||||
def restart(context: BaseComposeContext, services: t.List[str]) -> None:
|
||||
config = tutor_config.load(context.root)
|
||||
command = ["restart"]
|
||||
if "all" in services:
|
||||
@ -130,8 +227,14 @@ def restart(context: BaseComposeContext, services: List[str]) -> None:
|
||||
|
||||
@click.command(help="Initialise all applications")
|
||||
@click.option("-l", "--limit", help="Limit initialisation to this service or plugin")
|
||||
@mount_option
|
||||
@click.pass_obj
|
||||
def init(context: BaseComposeContext, limit: str) -> None:
|
||||
def init(
|
||||
context: BaseComposeContext,
|
||||
limit: str,
|
||||
mounts: t.Tuple[t.List[MountParam.MountType]],
|
||||
) -> None:
|
||||
process_mount_arguments(mounts)
|
||||
config = tutor_config.load(context.root)
|
||||
runner = context.job_runner(config)
|
||||
jobs.initialise(runner, limit_to=limit)
|
||||
@ -177,7 +280,9 @@ def createuser(
|
||||
)
|
||||
@click.argument("theme_name")
|
||||
@click.pass_obj
|
||||
def settheme(context: BaseComposeContext, domains: List[str], theme_name: str) -> None:
|
||||
def settheme(
|
||||
context: BaseComposeContext, domains: t.List[str], theme_name: str
|
||||
) -> None:
|
||||
config = tutor_config.load(context.root)
|
||||
runner = context.job_runner(config)
|
||||
domains = domains or jobs.get_all_openedx_domains(config)
|
||||
@ -202,9 +307,15 @@ def importdemocourse(context: BaseComposeContext) -> None:
|
||||
),
|
||||
context_settings={"ignore_unknown_options": True},
|
||||
)
|
||||
@mount_option
|
||||
@click.argument("args", nargs=-1, required=True)
|
||||
@click.pass_context
|
||||
def run(context: click.Context, args: List[str]) -> None:
|
||||
def run(
|
||||
context: click.Context,
|
||||
mounts: t.Tuple[t.List[MountParam.MountType]],
|
||||
args: t.List[str],
|
||||
) -> None:
|
||||
process_mount_arguments(mounts)
|
||||
extra_args = ["--rm"]
|
||||
if not utils.is_a_tty():
|
||||
extra_args.append("-T")
|
||||
@ -221,6 +332,9 @@ def run(context: click.Context, args: List[str]) -> None:
|
||||
@click.argument("path")
|
||||
@click.pass_obj
|
||||
def bindmount_command(context: BaseComposeContext, service: str, path: str) -> None:
|
||||
"""
|
||||
This command is made obsolete by the --mount arguments.
|
||||
"""
|
||||
config = tutor_config.load(context.root)
|
||||
host_path = bindmounts.create(context.job_runner(config), service, path)
|
||||
fmt.echo_info(
|
||||
@ -241,7 +355,7 @@ def bindmount_command(context: BaseComposeContext, service: str, path: str) -> N
|
||||
)
|
||||
@click.argument("args", nargs=-1, required=True)
|
||||
@click.pass_context
|
||||
def execute(context: click.Context, args: List[str]) -> None:
|
||||
def execute(context: click.Context, args: t.List[str]) -> None:
|
||||
context.invoke(dc_command, command="exec", args=args)
|
||||
|
||||
|
||||
@ -281,7 +395,7 @@ def status(context: click.Context) -> None:
|
||||
@click.argument("command")
|
||||
@click.argument("args", nargs=-1)
|
||||
@click.pass_obj
|
||||
def dc_command(context: BaseComposeContext, command: str, args: List[str]) -> None:
|
||||
def dc_command(context: BaseComposeContext, command: str, args: t.List[str]) -> None:
|
||||
config = tutor_config.load(context.root)
|
||||
volumes, non_volume_args = bindmounts.parse_volumes(args)
|
||||
volume_args = []
|
||||
@ -299,6 +413,72 @@ def dc_command(context: BaseComposeContext, command: str, args: List[str]) -> No
|
||||
context.job_runner(config).docker_compose(command, *volume_args, *non_volume_args)
|
||||
|
||||
|
||||
def process_mount_arguments(mounts: t.Tuple[t.List[MountParam.MountType]]) -> None:
|
||||
"""
|
||||
Process --mount arguments.
|
||||
|
||||
Most docker-compose commands support --mount arguments. This option
|
||||
is used to bind-mount folders from the host. A docker-compose.tmp.yml is
|
||||
generated at runtime and includes the bind-mounted volumes that were passed as CLI
|
||||
arguments.
|
||||
|
||||
Bind-mounts that are associated to "*-job" services will be added to the
|
||||
docker-compose jobs file.
|
||||
"""
|
||||
app_mounts: t.List[MountParam.MountType] = []
|
||||
job_mounts: t.List[MountParam.MountType] = []
|
||||
for mount in mounts:
|
||||
for service, host_path, container_path in mount:
|
||||
if service.endswith("-job"):
|
||||
job_mounts.append((service, host_path, container_path))
|
||||
else:
|
||||
app_mounts.append((service, host_path, container_path))
|
||||
|
||||
def _add_mounts(
|
||||
docker_compose: t.Dict[str, t.Any], bind_mounts: t.List[MountParam.MountType]
|
||||
) -> t.Dict[str, t.Any]:
|
||||
services = docker_compose.setdefault("services", {})
|
||||
for service, host_path, container_path in bind_mounts:
|
||||
fmt.echo_info(f"Bind-mount: {host_path} -> {container_path} in {service}")
|
||||
services.setdefault(service, {"volumes": []})
|
||||
services[service]["volumes"].append(f"{host_path}:{container_path}")
|
||||
return docker_compose
|
||||
|
||||
# Save bind-mounts
|
||||
@hooks.Filters.COMPOSE_LOCAL_TMP.add()
|
||||
def _add_mounts_to_docker_compose_tmp(
|
||||
docker_compose_tmp: t.Dict[str, t.Any]
|
||||
) -> t.Dict[str, t.Any]:
|
||||
return _add_mounts(docker_compose_tmp, app_mounts)
|
||||
|
||||
@hooks.Filters.COMPOSE_LOCAL_JOBS_TMP.add()
|
||||
def _add_mounts_to_docker_compose_jobs_tmp(
|
||||
docker_compose_tmp: t.Dict[str, t.Any]
|
||||
) -> t.Dict[str, t.Any]:
|
||||
return _add_mounts(docker_compose_tmp, job_mounts)
|
||||
|
||||
|
||||
@hooks.Filters.COMPOSE_MOUNTS.add()
|
||||
def _mount_edx_platform(
|
||||
volumes: t.List[t.Tuple[str, str]], name: str
|
||||
) -> t.List[t.Tuple[str, str]]:
|
||||
"""
|
||||
When mounting edx-platform with `--mount=/path/to/edx-platform`, bind-mount the host
|
||||
repo in the lms/cms containers.
|
||||
"""
|
||||
if name == "edx-platform":
|
||||
path = "/openedx/edx-platform"
|
||||
volumes += [
|
||||
("lms", path),
|
||||
("cms", path),
|
||||
("lms-worker", path),
|
||||
("cms-worker", path),
|
||||
("lms-job", path),
|
||||
("cms-job", path),
|
||||
]
|
||||
return volumes
|
||||
|
||||
|
||||
def add_commands(command_group: click.Group) -> None:
|
||||
command_group.add_command(start)
|
||||
command_group.add_command(stop)
|
||||
|
||||
@ -1,4 +1,4 @@
|
||||
from typing import List
|
||||
import typing as t
|
||||
|
||||
import click
|
||||
|
||||
@ -21,12 +21,14 @@ class DevJobRunner(compose.ComposeJobRunner):
|
||||
self.docker_compose_files += [
|
||||
tutor_env.pathjoin(self.root, "local", "docker-compose.yml"),
|
||||
tutor_env.pathjoin(self.root, "dev", "docker-compose.yml"),
|
||||
tutor_env.pathjoin(self.root, "local", "docker-compose.tmp.yml"),
|
||||
tutor_env.pathjoin(self.root, "local", "docker-compose.override.yml"),
|
||||
tutor_env.pathjoin(self.root, "dev", "docker-compose.override.yml"),
|
||||
]
|
||||
self.docker_compose_job_files += [
|
||||
tutor_env.pathjoin(self.root, "local", "docker-compose.jobs.yml"),
|
||||
tutor_env.pathjoin(self.root, "dev", "docker-compose.jobs.yml"),
|
||||
tutor_env.pathjoin(self.root, "local", "docker-compose.jobs.tmp.yml"),
|
||||
tutor_env.pathjoin(self.root, "local", "docker-compose.jobs.override.yml"),
|
||||
tutor_env.pathjoin(self.root, "dev", "docker-compose.jobs.override.yml"),
|
||||
]
|
||||
@ -99,21 +101,25 @@ Your Open edX platform is ready and can be accessed at the following urls:
|
||||
help="Run a development server",
|
||||
context_settings={"ignore_unknown_options": True},
|
||||
)
|
||||
@compose.mount_option
|
||||
@click.argument("options", nargs=-1, required=False)
|
||||
@click.argument("service")
|
||||
@click.pass_context
|
||||
def runserver(context: click.Context, options: List[str], service: str) -> None:
|
||||
def runserver(
|
||||
context: click.Context,
|
||||
mounts: t.Tuple[t.List[compose.MountParam.MountType]],
|
||||
options: t.List[str],
|
||||
service: str,
|
||||
) -> None:
|
||||
config = tutor_config.load(context.obj.root)
|
||||
if service in ["lms", "cms"]:
|
||||
port = 8000 if service == "lms" else 8001
|
||||
host = config["LMS_HOST"] if service == "lms" else config["CMS_HOST"]
|
||||
fmt.echo_info(
|
||||
"The {} service will be available at http://{}:{}".format(
|
||||
service, host, port
|
||||
)
|
||||
f"The {service} service will be available at http://{host}:{port}"
|
||||
)
|
||||
args = ["--service-ports", *options, service]
|
||||
context.invoke(compose.run, args=args)
|
||||
context.invoke(compose.run, mounts=mounts, args=args)
|
||||
|
||||
|
||||
dev.add_command(quickstart)
|
||||
|
||||
@ -174,7 +174,7 @@ def find_images_to_build(
|
||||
] = hooks.Filters.IMAGES_BUILD.iterate(config)
|
||||
found = False
|
||||
for name, path, tag, args in all_images_to_build:
|
||||
if name == image or image == "all":
|
||||
if image in [name, "all"]:
|
||||
found = True
|
||||
tag = tutor_env.render_str(config, tag)
|
||||
yield (name, path, tag, args)
|
||||
@ -196,7 +196,7 @@ def find_remote_image_tags(
|
||||
all_remote_images: t.Iterator[t.Tuple[str, str]] = filtre.iterate(config)
|
||||
found = False
|
||||
for name, tag in all_remote_images:
|
||||
if name == image or image == "all":
|
||||
if image in [name, "all"]:
|
||||
found = True
|
||||
yield tutor_env.render_str(config, tag)
|
||||
if not found:
|
||||
|
||||
@ -1,4 +1,4 @@
|
||||
from typing import Optional
|
||||
import typing as t
|
||||
|
||||
import click
|
||||
|
||||
@ -22,10 +22,12 @@ class LocalJobRunner(compose.ComposeJobRunner):
|
||||
self.docker_compose_files += [
|
||||
tutor_env.pathjoin(self.root, "local", "docker-compose.yml"),
|
||||
tutor_env.pathjoin(self.root, "local", "docker-compose.prod.yml"),
|
||||
tutor_env.pathjoin(self.root, "local", "docker-compose.tmp.yml"),
|
||||
tutor_env.pathjoin(self.root, "local", "docker-compose.override.yml"),
|
||||
]
|
||||
self.docker_compose_job_files += [
|
||||
tutor_env.pathjoin(self.root, "local", "docker-compose.jobs.yml"),
|
||||
tutor_env.pathjoin(self.root, "local", "docker-compose.jobs.tmp.yml"),
|
||||
tutor_env.pathjoin(self.root, "local", "docker-compose.jobs.override.yml"),
|
||||
]
|
||||
|
||||
@ -43,10 +45,16 @@ def local(context: click.Context) -> None:
|
||||
|
||||
|
||||
@click.command(help="Configure and run Open edX from scratch")
|
||||
@compose.mount_option
|
||||
@click.option("-I", "--non-interactive", is_flag=True, help="Run non-interactively")
|
||||
@click.option("-p", "--pullimages", is_flag=True, help="Update docker images")
|
||||
@click.pass_context
|
||||
def quickstart(context: click.Context, non_interactive: bool, pullimages: bool) -> None:
|
||||
def quickstart(
|
||||
context: click.Context,
|
||||
mounts: t.Tuple[t.List[compose.MountParam.MountType]],
|
||||
non_interactive: bool,
|
||||
pullimages: bool,
|
||||
) -> None:
|
||||
try:
|
||||
utils.check_macos_docker_memory()
|
||||
except exceptions.TutorError as e:
|
||||
@ -113,9 +121,9 @@ Press enter when you are ready to continue"""
|
||||
click.echo(fmt.title("Docker image updates"))
|
||||
context.invoke(compose.dc_command, command="pull")
|
||||
click.echo(fmt.title("Starting the platform in detached mode"))
|
||||
context.invoke(compose.start, detach=True)
|
||||
context.invoke(compose.start, mounts=mounts, detach=True)
|
||||
click.echo(fmt.title("Database creation and migrations"))
|
||||
context.invoke(compose.init)
|
||||
context.invoke(compose.init, mounts=mounts)
|
||||
|
||||
config = tutor_config.load(context.obj.root)
|
||||
fmt.echo_info(
|
||||
@ -142,7 +150,7 @@ Your Open edX platform is ready and can be accessed at the following urls:
|
||||
type=click.Choice(["ironwood", "juniper", "koa", "lilac"]),
|
||||
)
|
||||
@click.pass_context
|
||||
def upgrade(context: click.Context, from_release: Optional[str]) -> None:
|
||||
def upgrade(context: click.Context, from_release: t.Optional[str]) -> None:
|
||||
fmt.echo_alert(
|
||||
"This command only performs a partial upgrade of your Open edX platform. "
|
||||
"To perform a full upgrade, you should run `tutor local quickstart`."
|
||||
|
||||
@ -97,6 +97,55 @@ class Filters:
|
||||
return items
|
||||
"""
|
||||
|
||||
#: List of commands to be executed during initialization. These commands typically
|
||||
#: include database migrations, setting feature flags, etc.
|
||||
#:
|
||||
#: :parameter list[tuple[str, tuple[str, ...]]] tasks: list of ``(service, path)`` tasks.
|
||||
#:
|
||||
#: - ``service`` is the name of the container in which the task will be executed.
|
||||
#: - ``path`` is a tuple that corresponds to a template relative path.
|
||||
#: Example: ``("myplugin", "hooks", "myservice", "pre-init")`` (see:py:data:`IMAGES_BUILD`).
|
||||
#: The command to execute will be read from that template, after it is rendered.
|
||||
COMMANDS_INIT = filters.get("commands:init")
|
||||
|
||||
#: List of commands to be executed prior to initialization. These commands are run even
|
||||
#: before the mysql databases are created and the migrations are applied.
|
||||
#:
|
||||
#: :parameter list[tuple[str, tuple[str, ...]]] tasks: list of ``(service, path)`` tasks. (see :py:data:`COMMANDS_INIT`).
|
||||
COMMANDS_PRE_INIT = filters.get("commands:pre-init")
|
||||
|
||||
#: List of folders to bind-mount in docker-compose containers, either in ``tutor local`` or ``tutor dev``.
|
||||
#:
|
||||
#: Many ``tutor local`` and ``tutor dev`` commands support ``--mounts`` options
|
||||
#: that allow plugins to define custom behaviour at runtime. For instance
|
||||
#: ``--mount=/path/to/edx-platform`` would cause this host folder to be
|
||||
#: bind-mounted in different containers (lms, lms-worker, cms, cms-worker) at the
|
||||
#: /openedx/edx-platform location. Plugin developers may implement this filter to
|
||||
#: define custom behaviour when mounting folders that relate to their plugins. For
|
||||
#: instance, the ecommerce plugin may process the ``--mount=/path/to/ecommerce``
|
||||
#: option.
|
||||
#:
|
||||
#: :parameter list[tuple[str, str]] mounts: each item is a ``(service, path)``
|
||||
#: tuple, where ``service`` is the name of the docker-compose service and ``path`` is
|
||||
#: the location in the container where the folder should be bind-mounted. Note: the
|
||||
#: path must be slash-separated ("/"). Thus, do not use ``os.path.join`` to generate
|
||||
#: the ``path`` because it will fail on Windows.
|
||||
#: :parameter str name: basename of the host-mounted folder. In the example above,
|
||||
#: this is "edx-platform". When implementing this filter you should check this name to
|
||||
#: conditionnally add mounts.
|
||||
COMPOSE_MOUNTS = filters.get("compose:mounts")
|
||||
|
||||
#: Contents of the local/docker-compose.tmp.yml file that will be generated at
|
||||
#: runtime. This is used for instance to bind-mount folders from the host (see
|
||||
#: :py:data:`COMPOSE_MOUNTS`)
|
||||
#:
|
||||
#: :parameter dict[str, ...] docker_compose_tmp: values which will be serialized to local/docker-compose.tmp.yml.
|
||||
#: Keys and values will be rendered before saving, such that you may include ``{{ ... }}`` statements.
|
||||
COMPOSE_LOCAL_TMP = filters.get("compose:local:tmp")
|
||||
|
||||
#: Same as :py:data:`COMPOSE_LOCAL_TMP` but for jobs
|
||||
COMPOSE_LOCAL_JOBS_TMP = filters.get("compose:local-jobs:tmp")
|
||||
|
||||
#: List of images to be built when we run ``tutor images build ...``.
|
||||
#:
|
||||
#: :parameter list[tuple[str, tuple[str, ...], str, tuple[str, ...]]] tasks: list of ``(name, path, tag, args)`` tuples.
|
||||
@ -125,23 +174,6 @@ class Filters:
|
||||
#: Parameters are the same as for :py:data:`IMAGES_PULL`.
|
||||
IMAGES_PUSH = filters.get("images:push")
|
||||
|
||||
#: List of commands to be executed during initialization. These commands typically
|
||||
#: include database migrations, setting feature flags, etc.
|
||||
#:
|
||||
#: :parameter list[tuple[str, tuple[str, ...]]] tasks: list of ``(service, path)`` tasks.
|
||||
#:
|
||||
#: - ``service`` is the name of the container in which the task will be executed.
|
||||
#: - ``path`` is a tuple that corresponds to a template relative path.
|
||||
#: Example: ``("myplugin", "hooks", "myservice", "pre-init")`` (see:py:data:`IMAGES_BUILD`).
|
||||
#: The command to execute will be read from that template, after it is rendered.
|
||||
COMMANDS_INIT = filters.get("commands:init")
|
||||
|
||||
#: List of commands to be executed prior to initialization. These commands are run even
|
||||
#: before the mysql databases are created and the migrations are applied.
|
||||
#:
|
||||
#: :parameter list[tuple[str, tuple[str, ...]]] tasks: list of ``(service, path)`` tasks. (see :py:data:`COMMANDS_INIT`).
|
||||
COMMANDS_PRE_INIT = filters.get("commands:pre-init")
|
||||
|
||||
#: List of command line interface (CLI) commands.
|
||||
#:
|
||||
#: :parameter list commands: commands are instances of ``click.Command``. They will
|
||||
@ -209,6 +241,24 @@ class Filters:
|
||||
#: Note that Jinja2 filters are a completely different thing than the Tutor hook
|
||||
#: filters, although they share the same name.
|
||||
#:
|
||||
#: Out of the box, Tutor comes with the following filters:
|
||||
#:
|
||||
#: - ``common_domain``: Return the longest common name between two domain names. Example: ``{{ "studio.demo.myopenedx.com"|common_domain("lms.demo.myopenedx.com") }}`` is equal to "demo.myopenedx.com".
|
||||
#: - ``encrypt``: Encrypt an arbitrary string. The encryption process is compatible with `htpasswd <https://httpd.apache.org/docs/2.4/programs/htpasswd.html>`__ verification.
|
||||
#: - ``list_if``: In a list of ``(value, condition)`` tuples, return the list of ``value`` for which the ``condition`` is true.
|
||||
#: - ``long_to_base64``: Base-64 encode a long integer.
|
||||
#: - ``iter_values_named``: Yield the values of the configuration settings that match a certain pattern. Example: ``{% for value in iter_values_named(prefix="KEY", suffix="SUFFIX")%}...{% endfor %}``. By default, only non-empty values are yielded. To iterate also on empty values, pass the ``allow_empty=True`` argument.
|
||||
#: - ``patch``: See :ref:`patches <v0_plugin_patches>`.
|
||||
#: - ``random_string``: Return a random string of the given length composed of ASCII letters and digits. Example: ``{{ 8|random_string }}``.
|
||||
#: - ``reverse_host``: Reverse a domain name (see `reference <https://en.wikipedia.org/wiki/Reverse_domain_name_notation>`__). Example: ``{{ "demo.myopenedx.com"|reverse_host }}`` is equal to "com.myopenedx.demo".
|
||||
#: - ``rsa_import_key``: Import a PEM-formatted RSA key and return the corresponding object.
|
||||
#: - ``rsa_private_key``: Export an RSA private key in PEM format.
|
||||
#: - ``walk_templates``: Iterate recursively over the templates of the given folder. For instance::
|
||||
#:
|
||||
#: {% for file in "apps/myplugin"|walk_templates %}
|
||||
#: ...
|
||||
#: {% endfor %}
|
||||
#:
|
||||
#: :parameter filters: list of (name, function) tuples. The function signature
|
||||
#: should correspond to its usage in templates.
|
||||
ENV_TEMPLATE_FILTERS = filters.get("env:templates:filters")
|
||||
|
||||
@ -25,8 +25,9 @@ def _discover_entrypoint_plugins() -> None:
|
||||
Discover all plugins that declare a "tutor.plugin.v1" entrypoint.
|
||||
"""
|
||||
with hooks.Contexts.PLUGINS.enter():
|
||||
for entrypoint in pkg_resources.iter_entry_points("tutor.plugin.v1"):
|
||||
discover_package(entrypoint)
|
||||
if "TUTOR_IGNORE_ENTRYPOINT_PLUGINS" not in os.environ:
|
||||
for entrypoint in pkg_resources.iter_entry_points("tutor.plugin.v1"):
|
||||
discover_package(entrypoint)
|
||||
|
||||
|
||||
def discover_module(path: str) -> None:
|
||||
|
||||
@ -42,12 +42,12 @@ WORKDIR /openedx/edx-platform
|
||||
RUN git config --global user.email "tutor@overhang.io" \
|
||||
&& git config --global user.name "Tutor"
|
||||
|
||||
{% if patch("openedx-dockerfile-git-patches-default") %}
|
||||
{%- if patch("openedx-dockerfile-git-patches-default") %}
|
||||
# Custom edx-platform patches
|
||||
{{ patch("openedx-dockerfile-git-patches-default") }}
|
||||
{% else %}
|
||||
{%- else %}
|
||||
# Patch edx-platform
|
||||
{% endif %}
|
||||
{%- endif %}
|
||||
|
||||
{# Example: RUN git fetch --depth=2 https://github.com/openedx/edx-platform <GITSHA1> && git cherry-pick <GITSHA1> #}
|
||||
{{ patch("openedx-dockerfile-post-git-checkout") }}
|
||||
|
||||
@ -8,6 +8,7 @@ CMS_OAUTH2_KEY_SSO: "cms-sso"
|
||||
CMS_OAUTH2_KEY_SSO_DEV: "cms-sso-dev"
|
||||
CONTACT_EMAIL: "contact@{{ LMS_HOST }}"
|
||||
DEV_PROJECT_NAME: "{{ TUTOR_APP }}_dev"
|
||||
DOCKER_COMPOSE_VERSION: "3.7"
|
||||
DOCKER_REGISTRY: "docker.io/"
|
||||
DOCKER_IMAGE_OPENEDX: "{{ DOCKER_REGISTRY }}overhangio/openedx:{{ TUTOR_VERSION }}"
|
||||
DOCKER_IMAGE_OPENEDX_DEV: "openedx-dev"
|
||||
|
||||
@ -1,3 +1,3 @@
|
||||
version: "3.7"
|
||||
version: "{{ DOCKER_COMPOSE_VERSION }}"
|
||||
services: {% if not patch("dev-docker-compose-jobs-services") %}{}{% endif %}
|
||||
{{ patch("dev-docker-compose-jobs-services")|indent(4) }}
|
||||
|
||||
@ -1,4 +1,4 @@
|
||||
version: "3.7"
|
||||
version: "{{ DOCKER_COMPOSE_VERSION }}"
|
||||
|
||||
x-openedx-service:
|
||||
&openedx-service
|
||||
|
||||
@ -1,4 +1,4 @@
|
||||
version: "3.7"
|
||||
version: "{{ DOCKER_COMPOSE_VERSION }}"
|
||||
services:
|
||||
|
||||
mysql-job:
|
||||
|
||||
@ -1,4 +1,4 @@
|
||||
version: "3.7"
|
||||
version: "{{ DOCKER_COMPOSE_VERSION }}"
|
||||
services:
|
||||
# Web proxy for load balancing and SSL termination
|
||||
caddy:
|
||||
|
||||
@ -1,4 +1,4 @@
|
||||
version: "3.7"
|
||||
version: "{{ DOCKER_COMPOSE_VERSION }}"
|
||||
services:
|
||||
|
||||
############# External services
|
||||
|
||||
Loading…
Reference in New Issue
Block a user