Automatically bind-mount volumes from `volumes/`
This introduces a new dev/local command:
tutor dev bindmount CONTAINER PATH
And a new volume syntax:
tutor dev run --volume=PATH CONTAINER
This syntax automatically bind-mounts folders from the tutorroot/volumes
directory, which is pretty nifty.
This commit is contained in:
Régis Behmo
parent
b015890857
commit
1c927c6e96
|
|
@ -4,6 +4,8 @@ Note: Breaking changes between versions are indicated by "💥".
|
|||
|
||||
## Unreleased
|
||||
|
||||
- [Feature] Automatically bind-mount volumes from the `volumes/` directory with the `--volume=/...` syntax.
|
||||
|
||||
## v11.0.7 (2021-01-11)
|
||||
|
||||
- [Security] Apply security patch [26029](https://github.com/edx/edx-platform/pull/26029)
|
||||
|
|
@ -319,14 +321,14 @@ Note: Breaking changes between versions are indicated by "💥".
|
|||
|
||||
## 3.10.1 (2020-01-13)
|
||||
|
||||
- [Improvement] Explicitely point to docker.io images, when necessary, for [podman](https://podman.io/) compatibility
|
||||
- [Improvement] Explicitly point to docker.io images, when necessary, for [podman](https://podman.io/) compatibility
|
||||
|
||||
## 3.10.0 (2020-01-10)
|
||||
|
||||
- [Bugfix] Fix oauth authentication in dev mode
|
||||
- [Improvement] Upgrade to the 3.7 docker-compose syntax
|
||||
- [Improvement] The `dev runserver` command can now be run for just any service
|
||||
- 💥[Feature] `dev run/exec` commands now support generic options which are passed to docker-compose. Consequently, defining the `TUTOR_EDX_PLATFORM_PATH` environment variable no longer works. Instead, users are encouraged to explicitely pass the `-v` option, define a command alias or create a `docker-compose.override.yml` file.
|
||||
- 💥[Feature] `dev run/exec` commands now support generic options which are passed to docker-compose. Consequently, defining the `TUTOR_EDX_PLATFORM_PATH` environment variable no longer works. Instead, users are encouraged to explicitly pass the `-v` option, define a command alias or create a `docker-compose.override.yml` file.
|
||||
|
||||
## 3.9.1 (2020-01-08)
|
||||
|
||||
|
|
|
|||
83
docs/dev.rst
83
docs/dev.rst
|
|
@ -9,7 +9,7 @@ The following commands assume you have previously launched a :ref:`local <local>
|
|||
|
||||
tutor local quickstart
|
||||
|
||||
In order to run the platform in development mode, you **must** answer no ("n") to the question "Are you configuring a production platform".
|
||||
In order to run the platform in development mode, you **must** answer no ("n") to the question "Are you configuring a production platform".
|
||||
|
||||
Note that the local.overhang.io `domain <https://dnschecker.org/#A/local.overhang.io>`__ and its `subdomains <https://dnschecker.org/#CNAME/studio.local.overhang.io>`__ all point to 127.0.0.1. This is just a domain name that was setup to conveniently access a locally running Open edX platform.
|
||||
|
||||
|
|
@ -56,22 +56,46 @@ To collect assets, you can use the ``openedx-assets`` command that ships with Tu
|
|||
|
||||
tutor dev run lms openedx-assets --env=dev
|
||||
|
||||
Point to a local edx-platform
|
||||
-----------------------------
|
||||
.. _bind_mounts:
|
||||
|
||||
If you have one, you can point to a local version of `edx-platform <https://github.com/edx/edx-platform/>`_ on your host machine. To do so, pass a ``-v/--volume`` option to the ``run`` and runserver commands. For instance::
|
||||
Bind-mount container directories
|
||||
--------------------------------
|
||||
|
||||
tutor dev run -v /path/to/edx-platform:/openedx/edx-platform lms bash
|
||||
It may sometimes be convenient to mount container directories on the host, for instance: for editing and debugging. Tutor provides different solutions to this problem.
|
||||
|
||||
If you don't want to rewrite this option every time, you can define a command alias::
|
||||
Bind-mount from the "volumes/" directory
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
alias tutor-dev-run-lms="tutor dev run -v /path/to/edx-platform:/openedx/edx-platform lms"
|
||||
Tutor makes it easy to create a bind-mount from an existing container. First, copy the contents of a container directory with the ``bindmount`` command. For instance, to copy the virtual environment of the "lms" container::
|
||||
|
||||
For technical reasons, the ``-v`` option is only supported for the ``run`` and ``runserver`` commands. With these commands, only one service is started. But there are cases where you may want to launch and debug a complete Open edX platform with ``tutor dev start`` and mount a custom edx-platform fork. For instance, this might be needed when testing the interaction between multiple services. To do so, you should create a ``docker-compose.override.yml`` file that will specify a custom volume to be used with all ``dev`` commands::
|
||||
tutor dev bindmount lms /openedx/venv
|
||||
|
||||
This command recursively copies the contents of the ``/opendedx/venv`` directory to ``$(tutor config printroot)/volumes/venv``. The code of any Python dependency can then be edited -- for instance, you can then add a ``import ipdb; ipdb.set_trace()`` statement for step-by-step debugging, or implement a custom feature.
|
||||
|
||||
Then, bind-mount the directory back in the container with the ``--volume`` option::
|
||||
|
||||
tutor dev runserver --volume=/openedx/venv lms
|
||||
|
||||
Notice how the ``--volume=/openedx/venv`` option differs from `Docker syntax <https://docs.docker.com/storage/volumes/#choose-the--v-or---mount-flag>`__? Tutor recognizes this syntax and automatically converts this option to ``--volume=/path/to/tutor/root/volumes/venv:/openedx/venv``, which is recognized by Docker.
|
||||
|
||||
.. note::
|
||||
The ``bindmount`` command and the ``--volume=/...`` option syntax are available both for the ``tutor local`` and ``tutor dev`` commands.
|
||||
|
||||
Manual bind-mount to any directory
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The above solution may not work for you if you already have an existing directory, outside of the "volumes/" directory, which you would like mounted in one of your containers. For instance, you may want to mount your copy of the `edx-platform <https://github.com/edx/edx-platform/>`__ repository. In such cases, you can simply use the ``-v/--volume`` `Docker option <https://docs.docker.com/storage/volumes/#choose-the--v-or---mount-flag>`__::
|
||||
|
||||
tutor dev run --volume=/path/to/edx-platform:/openedx/edx-platform lms bash
|
||||
|
||||
Override docker-compose volumes
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The above solutions require that you explicitly pass the ``-v/--volume`` to every ``run`` or ``runserver`` command, which may be inconvenient. Also, these solutions are not compatible with the ``start`` command. To address these issues, you can create a ``docker-compose.override.yml`` file that will specify custom volumes to be used with all ``dev`` commands::
|
||||
|
||||
vim "$(tutor config printroot)/env/dev/docker-compose.override.yml"
|
||||
|
||||
Then, add the following content::
|
||||
You are then free to bind-mount any directory to any container. For instance, to mount your own edx-platform fork::
|
||||
|
||||
version: "3.7"
|
||||
services:
|
||||
|
|
@ -88,27 +112,54 @@ Then, add the following content::
|
|||
volumes:
|
||||
- /path/to/edx-platform/:/openedx/edx-platform
|
||||
|
||||
This override file will be loaded when running any ``tutor dev ..`` command. The edx-platform repo mounted at the specified path will be automaticall mounted inside all LMS and CMS containers. With this file, you should no longer specify the ``-v`` option from the command line with the ``run`` or ``runserver`` commands.
|
||||
This override file will be loaded when running any ``tutor dev ..`` command. The edx-platform repo mounted at the specified path will be automatically mounted inside all LMS and CMS containers. With this file, you should no longer specify the ``-v/--volume`` option from the command line with the ``run`` or ``runserver`` commands.
|
||||
|
||||
**Note:** containers are built on the Koa release. If you are working on a different version of Open edX, you will have to rebuild the ``openedx`` docker images with the version. See the :ref:`fork edx-platform section <edx_platform_fork>`.
|
||||
.. note::
|
||||
The ``tutor local`` commands loads the ``docker-compose.override.yml`` file from the ``$(tutor config printroot)/env/local/docker-compose.override.yml`` directory.
|
||||
|
||||
Point to a local edx-platform
|
||||
-----------------------------
|
||||
|
||||
Following the instructions :ref:`above <bind_mounts>` on how to bind-mount directories from the host above, you may mount your own `edx-platform <https://github.com/edx/edx-platform/>`__ fork in your containers by running either::
|
||||
|
||||
# Mount from the volumes/ directory
|
||||
tutor dev bindmount lms /openedx/edx-platform
|
||||
tutor dev runserver --volume=/openedx/edx-platform lms
|
||||
|
||||
# Mount from an arbitrary directory
|
||||
tutor dev runserver --volume=/path/to/edx-platform:/openedx/edx-platform lms
|
||||
|
||||
# Add your own volumes to $(tutor config printroot)/env/dev/docker-compose.override.yml
|
||||
tutor dev runserver lms
|
||||
|
||||
Prepare the edx-platform repo
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
In order to run a fork of edx-platform, dependencies need to be properly installed and assets compiled in that repo. To do so, run::
|
||||
If you choose any but the first solution above, you will have to make sure that your fork works with Tutor.
|
||||
|
||||
tutor dev run -v /path/to/edx-platform:/openedx/edx-platform lms bash
|
||||
First of all, you should make sure that you are working off the ``open-release/koa.1`` tag. See the :ref:`fork edx-platform section <edx_platform_fork>` for more information.
|
||||
|
||||
Then, you should run the following commands::
|
||||
|
||||
# Run bash in the lms container
|
||||
tutor dev run [--volume=...] lms bash
|
||||
|
||||
# Compile local python requirements
|
||||
pip install --requirement requirements/edx/development.txt
|
||||
python setup.py install
|
||||
|
||||
# Install nodejs packages in node_modules/
|
||||
npm install
|
||||
|
||||
# Rebuild static assets
|
||||
openedx-assets build --env=dev
|
||||
|
||||
|
||||
Debug edx-platform
|
||||
~~~~~~~~~~~~~~~~~~
|
||||
|
||||
To debug a local edx-platform repository, add a ``import ipdb; ipdb.set_trace()`` breakpoint anywhere in your code and run::
|
||||
|
||||
tutor dev runserver -v /path/to/edx-platform:/openedx/edx-platform lms
|
||||
tutor dev runserver [--volume=...] lms
|
||||
|
||||
XBlock and edx-platform plugin development
|
||||
------------------------------------------
|
||||
|
|
@ -184,4 +235,4 @@ You should then specify the settings to use on the host::
|
|||
|
||||
From then on, all ``dev`` commands will use the ``mysettings`` module. For instance::
|
||||
|
||||
tutor dev runserver -v /path/to/edx-platform:/openedx/edx-platform lms
|
||||
tutor dev runserver lms
|
||||
|
|
|
|||
|
|
@ -0,0 +1,31 @@
|
|||
import unittest
|
||||
|
||||
from tutor import bindmounts
|
||||
from tutor.exceptions import TutorError
|
||||
|
||||
|
||||
class BindMountsTests(unittest.TestCase):
|
||||
def test_get_name(self):
|
||||
self.assertEqual("venv", bindmounts.get_name("/openedx/venv"))
|
||||
self.assertEqual("venv", bindmounts.get_name("/openedx/venv/"))
|
||||
|
||||
def test_get_name_root_folder(self):
|
||||
with self.assertRaises(TutorError):
|
||||
bindmounts.get_name("/")
|
||||
with self.assertRaises(TutorError):
|
||||
bindmounts.get_name("")
|
||||
|
||||
def test_parse_volumes(self):
|
||||
volume_args, non_volume_args = bindmounts.parse_volumes(
|
||||
[
|
||||
"run",
|
||||
"--volume=/openedx/venv",
|
||||
"-v",
|
||||
"/tmp/openedx:/openedx",
|
||||
"lms",
|
||||
"echo",
|
||||
"boom",
|
||||
]
|
||||
)
|
||||
self.assertEqual(("/openedx/venv", "/tmp/openedx:/openedx"), volume_args)
|
||||
self.assertEqual(("run", "lms", "echo", "boom"), non_volume_args)
|
||||
|
|
@ -0,0 +1,74 @@
|
|||
import os
|
||||
|
||||
import click
|
||||
|
||||
from .exceptions import TutorError
|
||||
from .utils import get_user_id
|
||||
|
||||
|
||||
def create(root, config, docker_compose_func, service, path):
|
||||
volumes_root_path = get_root_path(root)
|
||||
volume_name = get_name(path)
|
||||
container_volumes_root_path = "/tmp/volumes"
|
||||
command = """rm -rf {volumes_path}/{volume_name}
|
||||
cp -r {src_path} {volumes_path}/{volume_name}
|
||||
chown -R {user_id} {volumes_path}/{volume_name}""".format(
|
||||
volumes_path=container_volumes_root_path,
|
||||
volume_name=volume_name,
|
||||
src_path=path,
|
||||
user_id=get_user_id(),
|
||||
)
|
||||
|
||||
# Create volumes root dir if it does not exist. Otherwise it is created with root owner and might not be writable
|
||||
# in the container, e.g: in the dev containers.
|
||||
if not os.path.exists(volumes_root_path):
|
||||
os.makedirs(volumes_root_path)
|
||||
|
||||
docker_compose_func(
|
||||
root,
|
||||
config,
|
||||
"run",
|
||||
"--rm",
|
||||
"--no-deps",
|
||||
"--volume",
|
||||
"{}:{}".format(volumes_root_path, container_volumes_root_path),
|
||||
service,
|
||||
"sh",
|
||||
"-e",
|
||||
"-c",
|
||||
command,
|
||||
)
|
||||
return os.path.join(volumes_root_path, volume_name)
|
||||
|
||||
|
||||
def get_path(root, container_bind_path):
|
||||
bind_basename = get_name(container_bind_path)
|
||||
return os.path.join(get_root_path(root), bind_basename)
|
||||
|
||||
|
||||
def get_name(container_bind_path):
|
||||
# We rstrip slashes, otherwise os.path.basename returns an empty string
|
||||
# We don't use basename here as it will not work on Windows
|
||||
name = container_bind_path.rstrip("/").split("/")[-1]
|
||||
if not name:
|
||||
raise TutorError("Mounting a container root folder is not supported")
|
||||
return name
|
||||
|
||||
|
||||
def get_root_path(root):
|
||||
return os.path.join(root, "volumes")
|
||||
|
||||
|
||||
def parse_volumes(args):
|
||||
"""
|
||||
Parse `-v/--volume` options from an arbitrary list of arguments.
|
||||
"""
|
||||
|
||||
@click.command(context_settings={"ignore_unknown_options": True})
|
||||
@click.option("-v", "--volume", "volumes", multiple=True)
|
||||
@click.argument("args", nargs=-1, required=True)
|
||||
def custom_docker_compose(volumes, args): # pylint: disable=unused-argument
|
||||
pass
|
||||
|
||||
context = custom_docker_compose.make_context("custom", args)
|
||||
return context.params["volumes"], context.params["args"]
|
||||
|
|
@ -1,7 +1,11 @@
|
|||
import os
|
||||
|
||||
import click
|
||||
|
||||
from .. import bindmounts
|
||||
from .. import config as tutor_config
|
||||
from .. import env as tutor_env
|
||||
from ..exceptions import TutorError
|
||||
from .. import fmt
|
||||
from .. import scripts
|
||||
from .. import serialize
|
||||
|
|
@ -180,22 +184,6 @@ def importdemocourse(context):
|
|||
scripts.import_demo_course(runner)
|
||||
|
||||
|
||||
@click.command(
|
||||
short_help="Direct interface to docker-compose.",
|
||||
help=(
|
||||
"Direct interface to docker-compose. This is a wrapper around `docker-compose`. All commands, options and"
|
||||
" arguments passed to this command will be forwarded to docker-compose."
|
||||
),
|
||||
context_settings={"ignore_unknown_options": True},
|
||||
name="dc",
|
||||
)
|
||||
@click.argument("args", nargs=-1, required=True)
|
||||
@click.pass_obj
|
||||
def dc_command(context, args):
|
||||
config = tutor_config.load(context.root)
|
||||
context.docker_compose(context.root, config, *args)
|
||||
|
||||
|
||||
@click.command(
|
||||
short_help="Run a command in a new container",
|
||||
help=(
|
||||
|
|
@ -207,10 +195,31 @@ def dc_command(context, args):
|
|||
)
|
||||
@click.argument("args", nargs=-1, required=True)
|
||||
def run(args):
|
||||
command = ["run", "--rm"]
|
||||
extra_args = ["--rm"]
|
||||
if not utils.is_a_tty():
|
||||
command.append("-T")
|
||||
dc_command.callback([*command, *args])
|
||||
extra_args.append("-T")
|
||||
dc_command.callback("run", [*extra_args, *args])
|
||||
|
||||
|
||||
@click.command(
|
||||
name="bindmount",
|
||||
help="Copy the contents of a container directory to a ready-to-bind-mount host directory",
|
||||
)
|
||||
@click.argument(
|
||||
"service",
|
||||
)
|
||||
@click.argument("path")
|
||||
@click.pass_obj
|
||||
def bindmount_command(context, service, path):
|
||||
config = tutor_config.load(context.root)
|
||||
host_path = bindmounts.create(
|
||||
context.root, config, context.docker_compose, service, path
|
||||
)
|
||||
fmt.echo_info(
|
||||
"Bind-mount volume created at {}. You can now use it in all `local` and `dev` commands with the `--volume={}` option.".format(
|
||||
host_path, path
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
@click.command(
|
||||
|
|
@ -225,7 +234,7 @@ def run(args):
|
|||
)
|
||||
@click.argument("args", nargs=-1, required=True)
|
||||
def execute(args):
|
||||
dc_command.callback(["exec", *args])
|
||||
dc_command.callback("exec", args)
|
||||
|
||||
|
||||
@click.command(
|
||||
|
|
@ -236,13 +245,47 @@ def execute(args):
|
|||
@click.option("--tail", type=int, help="Number of lines to show from each container")
|
||||
@click.argument("service", nargs=-1)
|
||||
def logs(follow, tail, service):
|
||||
command = ["logs"]
|
||||
args = []
|
||||
if follow:
|
||||
command += ["--follow"]
|
||||
args.append("--follow")
|
||||
if tail is not None:
|
||||
command += ["--tail", str(tail)]
|
||||
command += service
|
||||
dc_command.callback(command)
|
||||
args += ["--tail", str(tail)]
|
||||
args += service
|
||||
dc_command.callback("logs", args)
|
||||
|
||||
|
||||
@click.command(
|
||||
short_help="Direct interface to docker-compose.",
|
||||
help=(
|
||||
"Direct interface to docker-compose. This is a wrapper around `docker-compose`. Most commands, options and"
|
||||
" arguments passed to this command will be forwarded as-is to docker-compose."
|
||||
),
|
||||
context_settings={"ignore_unknown_options": True},
|
||||
name="dc",
|
||||
)
|
||||
@click.argument("command")
|
||||
@click.argument("args", nargs=-1, required=True)
|
||||
@click.pass_obj
|
||||
def dc_command(context, command, args):
|
||||
config = tutor_config.load(context.root)
|
||||
volumes, non_volume_args = bindmounts.parse_volumes(args)
|
||||
volume_args = []
|
||||
for volume_arg in volumes:
|
||||
if ":" not in volume_arg:
|
||||
# This is a bind-mounted volume from the "volumes/" folder.
|
||||
host_bind_path = bindmounts.get_path(context.root, volume_arg)
|
||||
if not os.path.exists(host_bind_path):
|
||||
raise TutorError(
|
||||
(
|
||||
"Bind-mount volume directory {} does not exist. It must first be created"
|
||||
" with the '{}' command."
|
||||
).format(host_bind_path, bindmount_command.name)
|
||||
)
|
||||
volume_arg = "{}:{}".format(host_bind_path, volume_arg)
|
||||
volume_args += ["--volume", volume_arg]
|
||||
context.docker_compose(
|
||||
context.root, config, command, *volume_args, *non_volume_args
|
||||
)
|
||||
|
||||
|
||||
def add_commands(command_group):
|
||||
|
|
@ -256,5 +299,6 @@ def add_commands(command_group):
|
|||
command_group.add_command(settheme)
|
||||
command_group.add_command(dc_command)
|
||||
command_group.add_command(run)
|
||||
command_group.add_command(bindmount_command)
|
||||
command_group.add_command(execute)
|
||||
command_group.add_command(logs)
|
||||
|
|
|
|||
|
|
@ -26,6 +26,6 @@ if [ -d /openedx/data/uploads/ ]; then
|
|||
fi
|
||||
fi
|
||||
|
||||
# Create waffle switches to enable some features, if they have not been explicitely defined before
|
||||
# Create waffle switches to enable some features, if they have not been explicitly defined before
|
||||
# Completion tracking: add green ticks to every completed unit
|
||||
(./manage.py lms waffle_switch --list | grep completion.enable_completion_tracking) || ./manage.py lms waffle_switch --create completion.enable_completion_tracking on
|
||||
|
|
|
|||
Loading…
Reference in New Issue