Better `dev` environment

The `dev` commands now rely on a different openedx-dev docker image.
This gives us multiple improvements:

- no more chown in base image
- faster chown in development
- mounted requirements volume in development
- fix static assets issues
- bundled ipdb/vim/... packages, which are convenient for development

Close #235

CHANGELOG.md

@@ -2,6 +2,11 @@
 
 Note: Breaking changes between versions are indicated by "💥".
 
+## Unreleased
+
+- 💥[Improvement] Better `dev` commands, with dedicated development docker image. One of the consequences is that the `dev watchthemes` command is replaced by `dev run lms watchthemes`.
+- [Improvement] `images` commands now accept multiple `image` arguments
+
 ## 3.7.4 (2019-10-19)
 
 - [Bugfix] Fix missing requirements file in pypi package (#261)

docs/dev.rst

@@ -15,6 +15,18 @@ Once the local platform has been configured, you should stop it so that it does
 
     tutor local stop
 
+Finally, you should build the ``openedx-dev`` docker image::
+
+    tutor images build openedx-dev
+
+This ``openedx-dev`` development image differs from the ``openedx`` production image:
+
+- The user that runs inside the container has the same UID as the user on the host, in order to avoid permission problems inside mounted volumes (and in particular in the edx-platform repository).
+- Additional python and system requirements are installed for convenient debugging: `ipython <https://ipython.org/>`__, `ipdb <https://pypi.org/project/ipdb/>`__, vim, telnet.
+- The edx-platform `development requirements <https://github.com/edx/edx-platform/blob/open-release/ironwood.2/requirements/edx/development.in>`__ are installed.
+
+Since the ``openedx-dev`` is based upon the ``openedx`` docker image, it should be re-built every time the ``openedx`` docker image is modified.
+
 Run a local development webserver
 ---------------------------------
 
@@ -71,7 +83,7 @@ In order to run a fork of edx-platform, dependencies need to be properly install
 Debug edx-platform
 ~~~~~~~~~~~~~~~~~~
 
-To debug a local edx-platform repository, add a ``import pdb; pdb.set_trace()`` breakpoint anywhere in your code and run::
+To debug a local edx-platform repository, add a ``import ipdb; ipdb.set_trace()`` breakpoint anywhere in your code and run::
 
     tutor dev runserver lms --edx-platform-path=/path/to/edx-platform
 
@@ -92,7 +104,7 @@ Then, follow the `Open edX documentation to apply your themes <https://edx.readt
 
 Watch the themes folders for changes (in a different terminal)::
 
-    tutor dev watchthemes
+    tutor dev run watchthemes
 
 Make changes to some of the files inside the theme directory: the theme assets should be automatically recompiled and visible at http://localhost:8000.
 

tests/openedx-lms-common-settings

@@ -0,0 +1 @@
+ORA2_FILEUPLOAD_BACKEND = "s3"

tests/test_images.py

@@ -0,0 +1,13 @@
+import unittest
+from tutor import images
+
+
+class ImagesTests(unittest.TestCase):
+    def test_get_tag(self):
+        config = {
+            "DOCKER_IMAGE_OPENEDX": "openedx",
+            "DOCKER_IMAGE_OPENEDX_DEV": "openedxdev",
+            "DOCKER_REGISTRY": "registry/",
+        }
+        self.assertEqual("registry/openedx", images.get_tag(config, "openedx"))
+        self.assertEqual("registry/openedxdev", images.get_tag(config, "openedx-dev"))

tutor/commands/dev.py

@@ -1,8 +1,7 @@
-import subprocess
-
 import click
 
 from .. import env as tutor_env
+from .. import fmt
 from .. import opts
 from .. import utils
 
@@ -28,10 +27,7 @@ def run(root, edx_platform_path, edx_platform_settings, service, command, args):
         run_command.append(command)
     if args:
         run_command += args
-    port = service_port(service)
-    docker_compose_run_with_port(
-        root, edx_platform_path, edx_platform_settings, port, *run_command
-    )
+    docker_compose_run(root, edx_platform_path, edx_platform_settings, *run_command)
 
 
 @click.command(
@@ -56,16 +52,16 @@ def execute(root, service, command, args):
 @click.argument("service", type=click.Choice(["lms", "cms"]))
 def runserver(root, edx_platform_path, edx_platform_settings, service):
     port = service_port(service)
-    from .. import fmt
 
     fmt.echo_info(
         "The {} service will be available at http://localhost:{}".format(service, port)
     )
-    docker_compose_run_with_port(
+    docker_compose_run(
         root,
         edx_platform_path,
         edx_platform_settings,
-        port,
+        "-p",
+        "{port}:{port}".format(port=port),
         service,
         "./manage.py",
         service,
@@ -80,53 +76,15 @@ def stop(root):
     docker_compose(root, "rm", "--stop", "--force")
 
 
-@click.command(help="Watch for changes in your themes and recompile assets when needed")
-@opts.root
-@opts.edx_platform_path
-@opts.edx_platform_settings
-def watchthemes(root, edx_platform_path, edx_platform_settings):
-    docker_compose_run(
-        root,
-        edx_platform_path,
-        edx_platform_settings,
-        "--no-deps",
-        "lms",
-        "openedx-assets",
-        "watch-themes",
-        "--env",
-        "dev",
-    )
-
-
-def docker_compose_run_with_port(
-    root, edx_platform_path, edx_platform_settings, port, *command
-):
-    docker_compose_run(
-        root,
-        edx_platform_path,
-        edx_platform_settings,
-        "-p",
-        "{port}:{port}".format(port=port),
-        *command
-    )
-
-
 def docker_compose_run(root, edx_platform_path, edx_platform_settings, *command):
     run_command = [
         "run",
         "--rm",
         "-e",
         "SETTINGS={}".format(edx_platform_settings),
-        "--volume={}:/openedx/themes".format(
-            tutor_env.pathjoin(root, "build", "openedx", "themes")
-        ),
     ]
     if edx_platform_path:
-        run_command += [
-            "--volume={}:/openedx/edx-platform".format(edx_platform_path),
-            "-e",
-            "USERID={}".format(subprocess.check_output(["id", "-u"]).strip().decode()),
-        ]
+        run_command += ["--volume={}:/openedx/edx-platform".format(edx_platform_path)]
     run_command += command
     docker_compose(root, *run_command)
 
@@ -135,6 +93,8 @@ def docker_compose(root, *command):
     return utils.docker_compose(
         "-f",
         tutor_env.pathjoin(root, "local", "docker-compose.yml"),
+        "-f",
+        tutor_env.pathjoin(root, "dev", "docker-compose.yml"),
         "--project-name",
         "tutor_dev",
         *command
@@ -149,4 +109,3 @@ dev.add_command(run)
 dev.add_command(execute, name="exec")
 dev.add_command(runserver)
 dev.add_command(stop)
-dev.add_command(watchthemes)

tutor/commands/images.py

@@ -1,3 +1,5 @@
+import subprocess
+
 import click
 
 from .. import config as tutor_config
@@ -6,7 +8,8 @@ from .. import images
 from .. import opts
 from .. import plugins
 
-OPENEDX_IMAGE_NAMES = ["openedx", "forum", "android"]
+BASE_IMAGE_NAMES = ["openedx", "forum", "android"]
+DEV_IMAGE_NAMES = ["openedx-dev"]
 
 
 @click.group(name="images", short_help="Manage docker images")
@@ -19,7 +22,7 @@ def images_command():
     help="Build the docker images necessary for an Open edX platform.",
 )
 @opts.root
-@click.argument("image")
+@click.argument("image", nargs=-1)
 @click.option(
     "--no-cache", is_flag=True, help="Do not use cache when building the image"
 )
@@ -31,11 +34,15 @@ def images_command():
 )
 def build(root, image, no_cache, build_arg):
     config = tutor_config.load(root)
+    for i in image:
+        build_image(root, config, i, no_cache, build_arg)
 
+
+def build_image(root, config, image, no_cache, build_arg):
     # Build base images
-    for img in OPENEDX_IMAGE_NAMES:
+    for img in BASE_IMAGE_NAMES:
         if image in [img, "all"]:
-            tag = get_tag(config, img)
+            tag = images.get_tag(config, img)
             images.build(
                 tutor_env.pathjoin(root, "build", img),
                 tag,
@@ -55,16 +62,34 @@ def build(root, image, no_cache, build_arg):
                     build_args=build_arg,
                 )
 
+    # Build dev images
+    user_id = subprocess.check_output(["id", "-u"]).strip().decode()
+    dev_build_args = list(build_arg) + ["USERID={}".format(user_id)]
+    for img in DEV_IMAGE_NAMES:
+        if image in [img, "all"]:
+            tag = images.get_tag(config, img)
+            images.build(
+                tutor_env.pathjoin(root, "build", img),
+                tag,
+                no_cache=no_cache,
+                build_args=dev_build_args,
+            )
+
 
 @click.command(short_help="Pull images from the Docker registry")
 @opts.root
-@click.argument("image")
+@click.argument("image", nargs=-1)
 def pull(root, image):
     config = tutor_config.load(root)
+    for i in image:
+        pull_image(config, i)
+
+
+def pull_image(config, image):
     # Pull base images
     for img in image_names(config):
         if image in [img, "all"]:
-            tag = get_tag(config, img)
+            tag = images.get_tag(config, img)
             images.pull(tag)
 
     # Pull plugin images
@@ -77,13 +102,17 @@ def pull(root, image):
 
 @click.command(short_help="Push images to the Docker registry")
 @opts.root
-@click.argument("image")
+@click.argument("image", nargs=-1)
 def push(root, image):
     config = tutor_config.load(root)
+    for i in image:
+        push_image(root, config, i)
+    
+def push_image(root, config, image):
     # Push base images
-    for img in OPENEDX_IMAGE_NAMES:
+    for img in BASE_IMAGE_NAMES:
         if image in [img, "all"]:
-            tag = get_tag(config, img)
+            tag = images.get_tag(config, img)
             images.push(tag)
 
     # Push plugin images
@@ -94,13 +123,8 @@ def push(root, image):
                 images.push(tag)
 
 
-def get_tag(config, name):
-    image = config["DOCKER_IMAGE_" + name.upper()]
-    return "{registry}{image}".format(registry=config["DOCKER_REGISTRY"], image=image)
-
-
 def image_names(config):
-    return OPENEDX_IMAGE_NAMES + vendor_image_names(config)
+    return BASE_IMAGE_NAMES + vendor_image_names(config)
 
 
 def vendor_image_names(config):

tutor/env.py

@@ -108,7 +108,7 @@ def render_full(root, config):
     """
     Render the full environment, including version information.
     """
-    for subdir in ["android", "apps", "build", "k8s", "local", "webui"]:
+    for subdir in ["android", "apps", "build", "dev", "k8s", "local", "webui"]:
         save_subdir(subdir, root, config)
     for plugin, path in plugins.iter_templates(config):
         save_plugin_templates(plugin, path, root, config)

tutor/images.py

@@ -2,6 +2,11 @@ from . import fmt
 from . import utils
 
 
+def get_tag(config, name):
+    image = config["DOCKER_IMAGE_" + name.upper().replace("-", "_")]
+    return "{registry}{image}".format(registry=config["DOCKER_REGISTRY"], image=image)
+
+
 def build(path, tag, no_cache=False, build_args=None):
     fmt.echo_info("Building image {}".format(tag))
     command = ["build", "-t", tag, path]

tutor/templates/apps/openedx/settings/partials/common_all.py

@@ -13,5 +13,33 @@ BULK_EMAIL_DEFAULT_FROM_EMAIL = "no-reply@" + ENV_TOKENS["LMS_BASE"]
 API_ACCESS_MANAGER_EMAIL = ENV_TOKENS["CONTACT_EMAIL"]
 API_ACCESS_FROM_EMAIL = ENV_TOKENS["CONTACT_EMAIL"]
 
+# Load module store settings from config files
+update_module_store_settings(MODULESTORE, doc_store_settings=DOC_STORE_CONFIG)
+
+# Set uploaded media file path
+MEDIA_ROOT = "/openedx/media/"
+
+# Video settings
+VIDEO_IMAGE_SETTINGS["STORAGE_KWARGS"]["location"] = MEDIA_ROOT
+VIDEO_TRANSCRIPTS_SETTINGS["STORAGE_KWARGS"]["location"] = MEDIA_ROOT
+
+# Change syslog-based loggers which don't work inside docker containers
+LOGGING["handlers"]["local"] = {
+    "class": "logging.handlers.WatchedFileHandler",
+    "filename": os.path.join(LOG_DIR, "all.log"),
+    "formatter": "standard",
+}
+LOGGING["handlers"]["tracking"] = {
+    "level": "DEBUG",
+    "class": "logging.handlers.WatchedFileHandler",
+    "filename": os.path.join(LOG_DIR, "tracking.log"),
+    "formatter": "standard",
+}
+LOGGING["loggers"]["tracking"]["handlers"] = ["console", "local", "tracking"]
+
+EMAIL_USE_SSL = {{ SMTP_USE_SSL }}
+
+LOCALE_PATHS.append("/openedx/locale")
+
 {{ patch("openedx-common-settings") }}
 ######## End of settings common to LMS and CMS

tutor/templates/apps/openedx/settings/partials/common_cms.py

@@ -2,33 +2,6 @@
 
 ######## Common CMS settings
 
-# Load module store settings from config files
-update_module_store_settings(MODULESTORE, doc_store_settings=DOC_STORE_CONFIG)
-
-# Set uploaded media file path
-MEDIA_ROOT = "/openedx/media/"
-
-# Video settings
-VIDEO_IMAGE_SETTINGS["STORAGE_KWARGS"]["location"] = MEDIA_ROOT
-VIDEO_TRANSCRIPTS_SETTINGS["STORAGE_KWARGS"]["location"] = MEDIA_ROOT
-
-# Change syslog-based loggers which don't work inside docker containers
-LOGGING["handlers"]["local"] = {
-    "class": "logging.handlers.WatchedFileHandler",
-    "filename": os.path.join(LOG_DIR, "all.log"),
-    "formatter": "standard",
-}
-LOGGING["handlers"]["tracking"] = {
-    "level": "DEBUG",
-    "class": "logging.handlers.WatchedFileHandler",
-    "filename": os.path.join(LOG_DIR, "tracking.log"),
-    "formatter": "standard",
-}
-LOGGING["loggers"]["tracking"]["handlers"] = ["console", "local", "tracking"]
-
-EMAIL_USE_SSL = {{ SMTP_USE_SSL }}
-
-LOCALE_PATHS.append("/openedx/locale")
 
 STUDIO_NAME = "{{ PLATFORM_NAME }} - Studio"
 

tutor/templates/apps/openedx/settings/partials/common_lms.py

@@ -2,32 +2,6 @@
 
 ######## Common LMS settings
 
-# Load module store settings from config files
-update_module_store_settings(MODULESTORE, doc_store_settings=DOC_STORE_CONFIG)
-
-# Set uploaded media file path
-MEDIA_ROOT = "/openedx/media/"
-
-# Video settings
-VIDEO_IMAGE_SETTINGS["STORAGE_KWARGS"]["location"] = MEDIA_ROOT
-VIDEO_TRANSCRIPTS_SETTINGS["STORAGE_KWARGS"]["location"] = MEDIA_ROOT
-
-# Change syslog-based loggers which don't work inside docker containers
-LOGGING["handlers"]["local"] = {
-    "class": "logging.handlers.WatchedFileHandler",
-    "filename": os.path.join(LOG_DIR, "all.log"),
-    "formatter": "standard",
-}
-LOGGING["handlers"]["tracking"] = {
-    "level": "DEBUG",
-    "class": "logging.handlers.WatchedFileHandler",
-    "filename": os.path.join(LOG_DIR, "tracking.log"),
-    "formatter": "standard",
-}
-LOGGING["loggers"]["tracking"]["handlers"] = ["console", "local", "tracking"]
-
-EMAIL_USE_SSL = {{ SMTP_USE_SSL }}
-
 # Fix media files paths
 VIDEO_IMAGE_SETTINGS["STORAGE_KWARGS"]["location"] = MEDIA_ROOT
 VIDEO_TRANSCRIPTS_SETTINGS["STORAGE_KWARGS"]["location"] = MEDIA_ROOT
@@ -47,7 +21,6 @@ GRADES_DOWNLOAD = {
     },
 }
 
-LOCALE_PATHS.append("/openedx/locale")
 
 # JWT is authentication for other openedx services
 JWT_AUTH["JWT_ISSUER"] = "{{ JWT_COMMON_ISSUER }}"

tutor/templates/build/openedx-dev/Dockerfile

@@ -0,0 +1,25 @@
+FROM {{ DOCKER_REGISTRY }}{{ DOCKER_IMAGE_OPENEDX }}
+MAINTAINER Overhang.io <contact@overhang.io>
+
+# Install useful system requirements
+RUN apt update && \
+    apt install -y vim telnet \
+    && rm -rf /var/lib/apt/lists/*
+
+# Install dev python requirements
+RUN pip install -r requirements/edx/development.txt
+RUN pip install ipdb==0.12.2 ipython==5.8.0
+
+# Configure new user
+ARG USERID=1000
+RUN useradd --home-dir /openedx --uid $USERID openedx
+RUN chown -R openedx:openedx /openedx
+
+# Copy new entrypoint (to take care of permission issues at runtime)
+COPY ./bin /openedx/bin
+RUN chmod a+x /openedx/bin/*
+
+# Default django settings
+ENV SETTINGS tutor.development
+
+# TODO recompile static assets and point to edx-platform

tutor/templates/build/openedx-dev/bin/docker-entrypoint.sh

@@ -0,0 +1,10 @@
+#!/bin/sh -e
+export DJANGO_SETTINGS_MODULE=$SERVICE_VARIANT.envs.$SETTINGS
+
+# Change file permissions of mounted volumes
+echo "Setting file permissions..."
+find /openedx -not -path "/openedx/edx-platform/*" -not -user openedx -exec chown openedx:openedx {} \+
+echo "File permissions set."
+
+# Run CMD as user openedx
+exec chroot --userspec="$openedx:openedx" --skip-chdir / env HOME=/openedx "$@"

tutor/templates/build/openedx/Dockerfile

@@ -44,7 +44,7 @@ RUN virtualenv /openedx/venv
 ENV PATH /openedx/venv/bin:${PATH}
 ENV VIRTUAL_ENV /openedx/venv/
 RUN pip install setuptools==39.0.1 pip==9.0.3
-RUN pip install -r requirements/edx/development.txt
+RUN pip install -r requirements/edx/base.txt
 
 # Install patched version of ora2
 RUN pip uninstall -y ora2 && \
@@ -107,7 +107,7 @@ ENV SETTINGS tutor.production
 
 {{ patch("openedx-dockerfile") }}
 
-# Entrypoint will fix permissions of all files and run commands as openedx
+# Entrypoint will set right environment variables
 ENTRYPOINT ["docker-entrypoint.sh"]
 
 # Run server

tutor/templates/build/openedx/bin/docker-entrypoint.sh

@@ -1,19 +1,3 @@
 #!/bin/sh -e
 export DJANGO_SETTINGS_MODULE=$SERVICE_VARIANT.envs.$SETTINGS
-USERID=${USERID:=0}
-
-## Configure user with a different USERID if requested.
-if [ "$USERID" -ne 0 ]
-    then
-        echo "creating new user 'openedx' with UID $USERID"
-        useradd --home-dir /openedx -u $USERID openedx
-
-        # Change file permissions
-        chown --no-dereference -R openedx /openedx
-
-        # Run CMD as different user
-        exec chroot --userspec="$USERID" --skip-chdir / env HOME=/openedx "$@"
-else
-        # Run CMD as root (business as usual)
-        exec "$@"
-fi
+exec "$@"

tutor/templates/config.yml

@@ -28,6 +28,7 @@ ANDROID_RELEASE_STORE_PASSWORD: "android store password"
 ANDROID_RELEASE_KEY_PASSWORD: "android release key password"
 ANDROID_RELEASE_KEY_ALIAS: "android release key alias"
 DOCKER_IMAGE_OPENEDX: "overhangio/openedx:{{ TUTOR_VERSION }}"
+DOCKER_IMAGE_OPENEDX_DEV: "overhangio/openedx-dev:{{ TUTOR_VERSION }}"
 DOCKER_IMAGE_ANDROID: "overhangio/openedx-android:{{ TUTOR_VERSION }}"
 DOCKER_IMAGE_FORUM: "overhangio/openedx-forum:{{ TUTOR_VERSION }}"
 DOCKER_IMAGE_MEMCACHED: "memcached:1.4.38"

tutor/templates/dev/docker-compose.yml

@@ -0,0 +1,29 @@
+version: "3"
+
+services:
+  x-openedx-service:
+    &openedx-service
+    image: {{ DOCKER_REGISTRY }}{{ DOCKER_IMAGE_OPENEDX_DEV }}
+    volumes:
+      # static assets
+      - ../../data/openedx/staticfiles:/openedx/staticfiles
+      # theme files
+      - ../build/openedx/themes:/openedx/themes
+      # editable requirements
+      - ../build/openedx/requirements:/openedx/requirements
+
+  lms:
+    <<: *openedx-service
+  cms:
+    <<: *openedx-service
+  lms_worker:
+    <<: *openedx-service
+  cms_worker:
+    <<: *openedx-service
+  
+  # Additional service for watching theme changes
+  watchthemes:
+    <<: *openedx-service
+    command: openedx-assets watch-themes --env dev
+  
+  {{ patch("local-docker-compose-dev-services")|indent(2) }}