christianlight/tutor
6
0
Fork 0
mirror of https://github.com/ChristianLight/tutor.git synced 2025-01-12 17:51:11 +00:00
Code Issues Releases Activity

Automatically bind-mount volumes from volumes/

Browse Source 
This introduces a new dev/local command:

    tutor dev bindmount CONTAINER PATH

And a new volume syntax:

    tutor dev run --volume=PATH CONTAINER

This syntax automatically bind-mounts folders from the tutorroot/volumes
directory, which is pretty nifty.
This commit is contained in:
Régis Behmo 2020-12-25 22:56:42 +01:00
parent b015890857
commit 1c927c6e96
6 changed files with 246 additions and 44 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
CHANGELOG.md
View File

@ -4,6 +4,8 @@ Note: Breaking changes between versions are indicated by "💥".
## Unreleased
- [Feature] Automatically bind-mount volumes from the `volumes/` directory with the `--volume=/...` syntax.
## v11.0.7 (2021-01-11)
- [Security] Apply security patch [26029](https://github.com/edx/edx-platform/pull/26029)
@ -319,14 +321,14 @@ Note: Breaking changes between versions are indicated by "💥".
## 3.10.1 (2020-01-13)
- [Improvement] Explicitely point to docker.io images, when necessary, for [podman](https://podman.io/) compatibility
- [Improvement] Explicitly point to docker.io images, when necessary, for [podman](https://podman.io/) compatibility
## 3.10.0 (2020-01-10)
- [Bugfix] Fix oauth authentication in dev mode
- [Improvement] Upgrade to the 3.7 docker-compose syntax
- [Improvement] The `dev runserver` command can now be run for just any service
- 💥[Feature] `dev run/exec` commands now support generic options which are passed to docker-compose. Consequently, defining the `TUTOR_EDX_PLATFORM_PATH` environment variable no longer works. Instead, users are encouraged to explicitely pass the `-v` option, define a command alias or create a `docker-compose.override.yml` file.
- 💥[Feature] `dev run/exec` commands now support generic options which are passed to docker-compose. Consequently, defining the `TUTOR_EDX_PLATFORM_PATH` environment variable no longer works. Instead, users are encouraged to explicitly pass the `-v` option, define a command alias or create a `docker-compose.override.yml` file.
## 3.9.1 (2020-01-08)

83
docs/dev.rst
View File

@ -9,7 +9,7 @@ The following commands assume you have previously launched a :ref:`local <local>
tutor local quickstart
In order to run the platform in development mode, you **must** answer no ("n") to the question "Are you configuring a production platform".
In order to run the platform in development mode, you **must** answer no ("n") to the question "Are you configuring a production platform".
Note that the local.overhang.io `domain <https://dnschecker.org/#A/local.overhang.io>`__ and its `subdomains <https://dnschecker.org/#CNAME/studio.local.overhang.io>`__ all point to 127.0.0.1. This is just a domain name that was setup to conveniently access a locally running Open edX platform.
@ -56,22 +56,46 @@ To collect assets, you can use the ``openedx-assets`` command that ships with Tu
tutor dev run lms openedx-assets --env=dev
Point to a local edx-platform
-----------------------------
.. _bind_mounts:
If you have one, you can point to a local version of `edx-platform <https://github.com/edx/edx-platform/>`_ on your host machine. To do so, pass a ``-v/--volume`` option to the ``run`` and runserver commands. For instance::
Bind-mount container directories
--------------------------------
tutor dev run -v /path/to/edx-platform:/openedx/edx-platform lms bash
It may sometimes be convenient to mount container directories on the host, for instance: for editing and debugging. Tutor provides different solutions to this problem.
If you don't want to rewrite this option every time, you can define a command alias::
Bind-mount from the "volumes/" directory
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
alias tutor-dev-run-lms="tutor dev run -v /path/to/edx-platform:/openedx/edx-platform lms"
Tutor makes it easy to create a bind-mount from an existing container. First, copy the contents of a container directory with the ``bindmount`` command. For instance, to copy the virtual environment of the "lms" container::
For technical reasons, the ``-v`` option is only supported for the ``run`` and ``runserver`` commands. With these commands, only one service is started. But there are cases where you may want to launch and debug a complete Open edX platform with ``tutor dev start`` and mount a custom edx-platform fork. For instance, this might be needed when testing the interaction between multiple services. To do so, you should create a ``docker-compose.override.yml`` file that will specify a custom volume to be used with all ``dev`` commands::
tutor dev bindmount lms /openedx/venv
This command recursively copies the contents of the ``/opendedx/venv`` directory to ``$(tutor config printroot)/volumes/venv``. The code of any Python dependency can then be edited -- for instance, you can then add a ``import ipdb; ipdb.set_trace()`` statement for step-by-step debugging, or implement a custom feature.
Then, bind-mount the directory back in the container with the ``--volume`` option::
tutor dev runserver --volume=/openedx/venv lms
Notice how the ``--volume=/openedx/venv`` option differs from `Docker syntax <https://docs.docker.com/storage/volumes/#choose-the--v-or---mount-flag>`__? Tutor recognizes this syntax and automatically converts this option to ``--volume=/path/to/tutor/root/volumes/venv:/openedx/venv``, which is recognized by Docker.
.. note::
The ``bindmount`` command and the ``--volume=/...`` option syntax are available both for the ``tutor local`` and ``tutor dev`` commands.
Manual bind-mount to any directory
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The above solution may not work for you if you already have an existing directory, outside of the "volumes/" directory, which you would like mounted in one of your containers. For instance, you may want to mount your copy of the `edx-platform <https://github.com/edx/edx-platform/>`__ repository. In such cases, you can simply use the ``-v/--volume`` `Docker option <https://docs.docker.com/storage/volumes/#choose-the--v-or---mount-flag>`__::
tutor dev run --volume=/path/to/edx-platform:/openedx/edx-platform lms bash
Override docker-compose volumes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The above solutions require that you explicitly pass the ``-v/--volume`` to every ``run`` or ``runserver`` command, which may be inconvenient. Also, these solutions are not compatible with the ``start`` command. To address these issues, you can create a ``docker-compose.override.yml`` file that will specify custom volumes to be used with all ``dev`` commands::
vim "$(tutor config printroot)/env/dev/docker-compose.override.yml"
Then, add the following content::
You are then free to bind-mount any directory to any container. For instance, to mount your own edx-platform fork::
version: "3.7"
services:
@ -88,27 +112,54 @@ Then, add the following content::
volumes:
- /path/to/edx-platform/:/openedx/edx-platform
This override file will be loaded when running any ``tutor dev ..`` command. The edx-platform repo mounted at the specified path will be automaticall mounted inside all LMS and CMS containers. With this file, you should no longer specify the ``-v`` option from the command line with the ``run`` or ``runserver`` commands.
This override file will be loaded when running any ``tutor dev ..`` command. The edx-platform repo mounted at the specified path will be automatically mounted inside all LMS and CMS containers. With this file, you should no longer specify the ``-v/--volume`` option from the command line with the ``run`` or ``runserver`` commands.
**Note:** containers are built on the Koa release. If you are working on a different version of Open edX, you will have to rebuild the ``openedx`` docker images with the version. See the :ref:`fork edx-platform section <edx_platform_fork>`.
.. note::
The ``tutor local`` commands loads the ``docker-compose.override.yml`` file from the ``$(tutor config printroot)/env/local/docker-compose.override.yml`` directory.
Point to a local edx-platform
-----------------------------
Following the instructions :ref:`above <bind_mounts>` on how to bind-mount directories from the host above, you may mount your own `edx-platform <https://github.com/edx/edx-platform/>`__ fork in your containers by running either::
# Mount from the volumes/ directory
tutor dev bindmount lms /openedx/edx-platform
tutor dev runserver --volume=/openedx/edx-platform lms
# Mount from an arbitrary directory
tutor dev runserver --volume=/path/to/edx-platform:/openedx/edx-platform lms
# Add your own volumes to $(tutor config printroot)/env/dev/docker-compose.override.yml
tutor dev runserver lms
Prepare the edx-platform repo
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In order to run a fork of edx-platform, dependencies need to be properly installed and assets compiled in that repo. To do so, run::
If you choose any but the first solution above, you will have to make sure that your fork works with Tutor.
tutor dev run -v /path/to/edx-platform:/openedx/edx-platform lms bash
First of all, you should make sure that you are working off the ``open-release/koa.1`` tag. See the :ref:`fork edx-platform section <edx_platform_fork>` for more information.
Then, you should run the following commands::
# Run bash in the lms container
tutor dev run [--volume=...] lms bash
# Compile local python requirements
pip install --requirement requirements/edx/development.txt
python setup.py install
# Install nodejs packages in node_modules/
npm install
# Rebuild static assets
openedx-assets build --env=dev
Debug edx-platform
~~~~~~~~~~~~~~~~~~
To debug a local edx-platform repository, add a ``import ipdb; ipdb.set_trace()`` breakpoint anywhere in your code and run::
tutor dev runserver -v /path/to/edx-platform:/openedx/edx-platform lms
tutor dev runserver [--volume=...] lms
XBlock and edx-platform plugin development
------------------------------------------
@ -184,4 +235,4 @@ You should then specify the settings to use on the host::
From then on, all ``dev`` commands will use the ``mysettings`` module. For instance::
tutor dev runserver -v /path/to/edx-platform:/openedx/edx-platform lms
tutor dev runserver lms

31
tests/test_bindmounts.py Normal file
View File

@ -0,0 +1,31 @@
import unittest
from tutor import bindmounts
from tutor.exceptions import TutorError
class BindMountsTests(unittest.TestCase):
def test_get_name(self):
self.assertEqual("venv", bindmounts.get_name("/openedx/venv"))
self.assertEqual("venv", bindmounts.get_name("/openedx/venv/"))
def test_get_name_root_folder(self):
with self.assertRaises(TutorError):
bindmounts.get_name("/")
with self.assertRaises(TutorError):
bindmounts.get_name("")
def test_parse_volumes(self):
volume_args, non_volume_args = bindmounts.parse_volumes(
[
"run",
"--volume=/openedx/venv",
"-v",
"/tmp/openedx:/openedx",
"lms",
"echo",
"boom",
]
)
self.assertEqual(("/openedx/venv", "/tmp/openedx:/openedx"), volume_args)
self.assertEqual(("run", "lms", "echo", "boom"), non_volume_args)

74
tutor/bindmounts.py Normal file
View File

@ -0,0 +1,74 @@
import os
import click
from .exceptions import TutorError
from .utils import get_user_id
def create(root, config, docker_compose_func, service, path):
volumes_root_path = get_root_path(root)
volume_name = get_name(path)
container_volumes_root_path = "/tmp/volumes"
command = """rm -rf {volumes_path}/{volume_name}
cp -r {src_path} {volumes_path}/{volume_name}
chown -R {user_id} {volumes_path}/{volume_name}""".format(
volumes_path=container_volumes_root_path,
volume_name=volume_name,
src_path=path,
user_id=get_user_id(),
)
# Create volumes root dir if it does not exist. Otherwise it is created with root owner and might not be writable
# in the container, e.g: in the dev containers.
if not os.path.exists(volumes_root_path):
os.makedirs(volumes_root_path)
docker_compose_func(
root,
config,
"run",
"--rm",
"--no-deps",
"--volume",
"{}:{}".format(volumes_root_path, container_volumes_root_path),
service,
"sh",
"-e",
"-c",
command,
)
return os.path.join(volumes_root_path, volume_name)
def get_path(root, container_bind_path):
bind_basename = get_name(container_bind_path)
return os.path.join(get_root_path(root), bind_basename)
def get_name(container_bind_path):
# We rstrip slashes, otherwise os.path.basename returns an empty string
# We don't use basename here as it will not work on Windows
name = container_bind_path.rstrip("/").split("/")[-1]
if not name:
raise TutorError("Mounting a container root folder is not supported")
return name
def get_root_path(root):
return os.path.join(root, "volumes")
def parse_volumes(args):
"""
Parse `-v/--volume` options from an arbitrary list of arguments.
"""
@click.command(context_settings={"ignore_unknown_options": True})
@click.option("-v", "--volume", "volumes", multiple=True)
@click.argument("args", nargs=-1, required=True)
def custom_docker_compose(volumes, args): # pylint: disable=unused-argument
pass
context = custom_docker_compose.make_context("custom", args)
return context.params["volumes"], context.params["args"]

94
tutor/commands/compose.py
View File

@ -1,7 +1,11 @@
import os
import click
from .. import bindmounts
from .. import config as tutor_config
from .. import env as tutor_env
from ..exceptions import TutorError
from .. import fmt
from .. import scripts
from .. import serialize
@ -180,22 +184,6 @@ def importdemocourse(context):
scripts.import_demo_course(runner)
@click.command(
short_help="Direct interface to docker-compose.",
help=(
"Direct interface to docker-compose. This is a wrapper around `docker-compose`. All commands, options and"
" arguments passed to this command will be forwarded to docker-compose."
),
context_settings={"ignore_unknown_options": True},
name="dc",
)
@click.argument("args", nargs=-1, required=True)
@click.pass_obj
def dc_command(context, args):
config = tutor_config.load(context.root)
context.docker_compose(context.root, config, *args)
@click.command(
short_help="Run a command in a new container",
help=(
@ -207,10 +195,31 @@ def dc_command(context, args):
)
@click.argument("args", nargs=-1, required=True)
def run(args):
command = ["run", "--rm"]
extra_args = ["--rm"]
if not utils.is_a_tty():
command.append("-T")
dc_command.callback([*command, *args])
extra_args.append("-T")
dc_command.callback("run", [*extra_args, *args])
@click.command(
name="bindmount",
help="Copy the contents of a container directory to a ready-to-bind-mount host directory",
)
@click.argument(
"service",
)
@click.argument("path")
@click.pass_obj
def bindmount_command(context, service, path):
config = tutor_config.load(context.root)
host_path = bindmounts.create(
context.root, config, context.docker_compose, service, path
)
fmt.echo_info(
"Bind-mount volume created at {}. You can now use it in all `local` and `dev` commands with the `--volume={}` option.".format(
host_path, path
)
)
@click.command(
@ -225,7 +234,7 @@ def run(args):
)
@click.argument("args", nargs=-1, required=True)
def execute(args):
dc_command.callback(["exec", *args])
dc_command.callback("exec", args)
@click.command(
@ -236,13 +245,47 @@ def execute(args):
@click.option("--tail", type=int, help="Number of lines to show from each container")
@click.argument("service", nargs=-1)
def logs(follow, tail, service):
command = ["logs"]
args = []
if follow:
command += ["--follow"]
args.append("--follow")
if tail is not None:
command += ["--tail", str(tail)]
command += service
dc_command.callback(command)
args += ["--tail", str(tail)]
args += service
dc_command.callback("logs", args)
@click.command(
short_help="Direct interface to docker-compose.",
help=(
"Direct interface to docker-compose. This is a wrapper around `docker-compose`. Most commands, options and"
" arguments passed to this command will be forwarded as-is to docker-compose."
),
context_settings={"ignore_unknown_options": True},
name="dc",
)
@click.argument("command")
@click.argument("args", nargs=-1, required=True)
@click.pass_obj
def dc_command(context, command, args):
config = tutor_config.load(context.root)
volumes, non_volume_args = bindmounts.parse_volumes(args)
volume_args = []
for volume_arg in volumes:
if ":" not in volume_arg:
# This is a bind-mounted volume from the "volumes/" folder.
host_bind_path = bindmounts.get_path(context.root, volume_arg)
if not os.path.exists(host_bind_path):
raise TutorError(
(
"Bind-mount volume directory {} does not exist. It must first be created"
" with the '{}' command."
).format(host_bind_path, bindmount_command.name)
)
volume_arg = "{}:{}".format(host_bind_path, volume_arg)
volume_args += ["--volume", volume_arg]
context.docker_compose(
context.root, config, command, *volume_args, *non_volume_args
)
def add_commands(command_group):
@ -256,5 +299,6 @@ def add_commands(command_group):
command_group.add_command(settheme)
command_group.add_command(dc_command)
command_group.add_command(run)
command_group.add_command(bindmount_command)
command_group.add_command(execute)
command_group.add_command(logs)

2
tutor/templates/hooks/lms/init
View File

@ -26,6 +26,6 @@ if [ -d /openedx/data/uploads/ ]; then
fi
fi
# Create waffle switches to enable some features, if they have not been explicitely defined before
# Create waffle switches to enable some features, if they have not been explicitly defined before
# Completion tracking: add green ticks to every completed unit
(./manage.py lms waffle_switch --list | grep completion.enable_completion_tracking) || ./manage.py lms waffle_switch --create completion.enable_completion_tracking on