Merge remote-tracking branch 'origin/master' into nightly

CHANGELOG.md

@@ -4,6 +4,9 @@ Note: Breaking changes between versions are indicated by "💥".
 
 ## Unreleased
 
+- [Feature] Add `tutor [dev|local|k8s] status` command, which provides basic information about the platform's status.
+- 💥[Improvement] Make it possible to run `tutor k8s exec <command with multiple arguments>` (#636). As a consequence, it is no longer possible to run quoted commands: `tutor k8s exec "<some command>"`. Instead, you should remove the quotes: `tutor k8s exec <some command>`.
+
 ## v13.1.11 (2022-04-12)
 
 - [Security] Apply SAML security fix.

docs/_release_description.md

@@ -1,6 +1,6 @@
 Install this version from pip with::
 
-    pip install tutor[full]==TUTOR_VERSION
+    pip install "tutor[full]==TUTOR_VERSION"
 
 Or download the compiled binaries::
 

docs/configuration.rst

@@ -87,6 +87,10 @@ This defines the git repository from which you install Open edX platform code. I
 
 This defines the default version that will be pulled from all Open edX git repositories.
 
+- ``EDX_PLATFORM_VERSION`` (default: the value of ``OPENEDX_COMMON_VERSION``)
+
+This defines the version that will be pulled from just the Open edX platform git repositories. You may also override this configuration parameter at build time, by providing a ``--build-arg`` option.
+
 - ``OPENEDX_CMS_UWSGI_WORKERS`` (default: ``2``)
 - ``OPENEDX_LMS_UWSGI_WORKERS`` (default: ``2``)
 
@@ -222,7 +226,7 @@ openedx Docker Image build arguments
 When building the "openedx" Docker image, it is possible to specify a few `arguments <https://docs.docker.com/engine/reference/builder/#arg>`__:
 
 - ``EDX_PLATFORM_REPOSITORY`` (default: ``"{{ EDX_PLATFORM_REPOSITORY }}"``)
-- ``EDX_PLATFORM_VERSION`` (default: ``"{{ OPENEDX_COMMON_VERSION }}"``)
+- ``EDX_PLATFORM_VERSION`` (default: ``"{{ EDX_PLATFORM_VERSION }}"``, which if unset defaults to ``{{ OPENEDX_COMMON_VERSION }}``)
 - ``NPM_REGISTRY`` (default: ``"{{ NPM_REGISTRY }}"``)
 
 These arguments can be specified from the command line, `very much like Docker <https://docs.docker.com/engine/reference/commandline/build/#set-build-time-variables---build-arg>`__. For instance::

docs/download/pip.rst

@@ -1,3 +1,3 @@
 .. parsed-literal::
 
-    pip install tutor[full]
+    pip install "tutor[full]"

docs/install.rst

@@ -89,7 +89,7 @@ Upgrading
 
 To upgrade Open edX or benefit from the latest features and bug fixes, you should simply upgrade Tutor. Start by upgrading the "tutor" package and its dependencies::
 
-    pip install --upgrade tutor[full]
+    pip install --upgrade "tutor[full]"
 
 Then run the ``quickstart`` command again. Depending on your deployment target, run either::
 

docs/k8s.rst

@@ -64,6 +64,10 @@ On Minikube, the dashboard is already installed. To access the dashboard, run::
 
     minikube dashboard
 
+Lastly, Tutor itself provides a rudimentary listing of your cluster's nodes, workloads, and services::
+
+    tutor k8s status
+
 Technical details
 -----------------
 

docs/local.rst

@@ -98,6 +98,14 @@ Finally, tracking logs that store `user events <https://edx.readthedocs.io/proje
     $(tutor config printroot)/data/lms/logs/tracking.log
     $(tutor config printroot)/data/cms/logs/tracking.log
 
+Status
+~~~~~~
+
+You can view your platform's containers::
+
+    tutor local status
+
+Notice the **State** column in the output. It will tell you whether each container is starting, restarting, running (``Up``), cleanly stopped (``Exit 0``), or stopped on error (``Exit N``, where N ≠ 0).
 
 Common tasks
 ------------

docs/tutorials/nightly.rst

@@ -11,16 +11,16 @@ Installing Tutor Nightly
 Running Tutor Nightly requires more than setting a few configuration variables: because there are so many Open edX settings, version numbers, etc. which may change between the latest release and the current master branch, Tutor Nightly is actually maintained as a separate branch of the Tutor repository. To install Tutor Nightly, you should install Tutor from the "nightly" branch of the source repository. To do so, run::
 
     git clone --branch=nightly https://github.com/overhangio/tutor.git
-    pip install -e ./tutor
+    pip install -e "./tutor[full]"
 
 As usual, it is strongly recommended to run the command above in a `Python virtual environment <https://docs.python.org/3/tutorial/venv.html>`__.
 
-All Tutor plugins that you wish to use should likewise be installed from the "nightly branch". For instance, the `MFE plugin <https://github.com/overhangio/tutor-mfe>`__::
+In addition to installing Tutor Nightly itself, this will install automatically the nightly versions of all official Tutor plugins (which are enumerated in `plugins.txt <https://github.com/overhangio/tutor/tree/nightly/requirements/plugins.txt>`_). Alternatively, if you wish to hack on an official plugin or install a custom plugin, you can clone that plugin's repository and install it. For instance::
 
-    git clone --branch=nightly https://github.com/overhangio/tutor-mfe.git
+    git clone --branch=nightly https://github.com/myorganization/tutor-contrib-myplugin.git
-    pip install -e ./tutor-mfe
+    pip install -e ./tutor-contrib-myplugin
 
-You can then run the usual ``tutor`` commands::
+Once Tutor Nightly is installed, you can run the usual ``tutor`` commands::
 
     tutor local quickstart
     tutor local stop

tutor/commands/compose.py

@@ -263,6 +263,12 @@ def logs(context: click.Context, follow: bool, tail: bool, service: str) -> None
     context.invoke(dc_command, command="logs", args=args)
 
 
+@click.command(help="Print status information for containers")
+@click.pass_context
+def status(context: click.Context) -> None:
+    context.invoke(dc_command, command="ps")
+
+
 @click.command(
     short_help="Direct interface to docker-compose.",
     help=(
@@ -307,3 +313,4 @@ def add_commands(command_group: click.Group) -> None:
     command_group.add_command(bindmount_command)
     command_group.add_command(execute)
     command_group.add_command(logs)
+    command_group.add_command(status)

tutor/commands/k8s.py

@@ -8,7 +8,7 @@ from tutor import config as tutor_config
 from tutor import env as tutor_env
 from tutor import exceptions, fmt, jobs, serialize, utils
 from tutor.commands.config import save as config_save_command
-from tutor.commands.context import Context
+from tutor.commands.context import BaseJobContext
 from tutor.commands.upgrade.k8s import upgrade_from
 from tutor.types import Config, get_typed
 
@@ -147,9 +147,15 @@ class K8sJobRunner(jobs.BaseJobRunner):
         return 0
 
 
+class K8sContext(BaseJobContext):
+    def job_runner(self, config: Config) -> K8sJobRunner:
+        return K8sJobRunner(self.root, config)
+
+
 @click.group(help="Run Open edX on Kubernetes")
-def k8s() -> None:
+@click.pass_context
-    pass
+def k8s(context: click.Context) -> None:
+    context.obj = K8sContext(context.obj.root)
 
 
 @click.command(help="Configure and run Open edX from scratch")
@@ -213,7 +219,7 @@ Press enter when you are ready to continue"""
 )
 @click.argument("names", metavar="name", nargs=-1)
 @click.pass_obj
-def start(context: Context, names: List[str]) -> None:
+def start(context: K8sContext, names: List[str]) -> None:
     config = tutor_config.load(context.root)
     # Create namespace, if necessary
     # Note that this step should not be run for some users, in particular those
@@ -263,7 +269,7 @@ def start(context: Context, names: List[str]) -> None:
 )
 @click.argument("names", metavar="name", nargs=-1)
 @click.pass_obj
-def stop(context: Context, names: List[str]) -> None:
+def stop(context: K8sContext, names: List[str]) -> None:
     config = tutor_config.load(context.root)
     names = names or ["all"]
     for name in names:
@@ -301,7 +307,7 @@ def reboot(context: click.Context) -> None:
 @click.command(help="Completely delete an existing platform")
 @click.option("-y", "--yes", is_flag=True, help="Do not ask for confirmation")
 @click.pass_obj
-def delete(context: Context, yes: bool) -> None:
+def delete(context: K8sContext, yes: bool) -> None:
     if not yes:
         click.confirm(
             "Are you sure you want to delete the platform? All data will be removed.",
@@ -319,9 +325,9 @@ def delete(context: Context, yes: bool) -> None:
 @click.command(help="Initialise all applications")
 @click.option("-l", "--limit", help="Limit initialisation to this service or plugin")
 @click.pass_obj
-def init(context: Context, limit: Optional[str]) -> None:
+def init(context: K8sContext, limit: Optional[str]) -> None:
     config = tutor_config.load(context.root)
-    runner = K8sJobRunner(context.root, config)
+    runner = context.job_runner(config)
     wait_for_pod_ready(config, "caddy")
     for name in ["elasticsearch", "mysql", "mongodb"]:
         if tutor_config.is_service_activated(config, name):
@@ -333,7 +339,7 @@ def init(context: Context, limit: Optional[str]) -> None:
 @click.argument("deployment")
 @click.argument("replicas", type=int)
 @click.pass_obj
-def scale(context: Context, deployment: str, replicas: int) -> None:
+def scale(context: K8sContext, deployment: str, replicas: int) -> None:
     config = tutor_config.load(context.root)
     utils.kubectl(
         "scale",
@@ -354,25 +360,32 @@ def scale(context: Context, deployment: str, replicas: int) -> None:
     "-p",
     "--password",
     help="Specify password from the command line. If undefined, you will be prompted to input a password",
+    prompt=True,
+    hide_input=True,
 )
 @click.argument("name")
 @click.argument("email")
 @click.pass_obj
 def createuser(
-    context: Context, superuser: str, staff: bool, password: str, name: str, email: str
+    context: K8sContext,
+    superuser: str,
+    staff: bool,
+    password: str,
+    name: str,
+    email: str,
 ) -> None:
     config = tutor_config.load(context.root)
     command = jobs.create_user_command(superuser, staff, name, email, password=password)
-    # This needs to be interactive in case the user needs to type a password
+    runner = context.job_runner(config)
-    kubectl_exec(config, "lms", command, attach=True)
+    runner.run_job("lms", command)
 
 
 @click.command(help="Import the demo course")
 @click.pass_obj
-def importdemocourse(context: Context) -> None:
+def importdemocourse(context: K8sContext) -> None:
     fmt.echo_info("Importing demo course")
     config = tutor_config.load(context.root)
-    runner = K8sJobRunner(context.root, config)
+    runner = context.job_runner(config)
     jobs.import_demo_course(runner)
 
 
@@ -391,20 +404,24 @@ def importdemocourse(context: Context) -> None:
 )
 @click.argument("theme_name")
 @click.pass_obj
-def settheme(context: Context, domains: List[str], theme_name: str) -> None:
+def settheme(context: K8sContext, domains: List[str], theme_name: str) -> None:
     config = tutor_config.load(context.root)
-    runner = K8sJobRunner(context.root, config)
+    runner = context.job_runner(config)
     domains = domains or jobs.get_all_openedx_domains(config)
     jobs.set_theme(theme_name, domains, runner)
 
 
-@click.command(name="exec", help="Execute a command in a pod of the given application")
+@click.command(
+    name="exec",
+    help="Execute a command in a pod of the given application",
+    context_settings={"ignore_unknown_options": True},
+)
 @click.argument("service")
-@click.argument("command")
+@click.argument("args", nargs=-1, required=True)
 @click.pass_obj
-def exec_command(context: Context, service: str, command: str) -> None:
+def exec_command(context: K8sContext, service: str, args: List[str]) -> None:
     config = tutor_config.load(context.root)
-    kubectl_exec(config, service, command, attach=True)
+    kubectl_exec(config, service, args)
 
 
 @click.command(help="View output from containers")
@@ -414,7 +431,7 @@ def exec_command(context: Context, service: str, command: str) -> None:
 @click.argument("service")
 @click.pass_obj
 def logs(
-    context: Context, container: str, follow: bool, tail: bool, service: str
+    context: K8sContext, container: str, follow: bool, tail: bool, service: str
 ) -> None:
     config = tutor_config.load(context.root)
 
@@ -435,7 +452,7 @@ def logs(
 @click.command(help="Wait for a pod to become ready")
 @click.argument("name")
 @click.pass_obj
-def wait(context: Context, name: str) -> None:
+def wait(context: K8sContext, name: str) -> None:
     config = tutor_config.load(context.root)
     wait_for_pod_ready(config, name)
 
@@ -476,7 +493,7 @@ def upgrade(context: click.Context, from_release: Optional[str]) -> None:
 )
 @click.argument("args", nargs=-1)
 @click.pass_obj
-def apply_command(context: Context, args: List[str]) -> None:
+def apply_command(context: K8sContext, args: List[str]) -> None:
     kubectl_apply(context.root, *args)
 
 
@@ -484,9 +501,14 @@ def kubectl_apply(root: str, *args: str) -> None:
     utils.kubectl("apply", "--kustomize", tutor_env.pathjoin(root), *args)
 
 
-def kubectl_exec(
+@click.command(help="Print status information for all k8s resources")
-    config: Config, service: str, command: str, attach: bool = False
+@click.pass_obj
-) -> int:
+def status(context: K8sContext) -> int:
+    config = tutor_config.load(context.root)
+    return utils.kubectl("get", "all", *resource_namespace_selector(config))
+
+
+def kubectl_exec(config: Config, service: str, command: List[str]) -> int:
     selector = f"app.kubernetes.io/name={service}"
     pods = K8sClients.instance().core_api.list_namespaced_pod(
         namespace=k8s_namespace(config), label_selector=selector
@@ -498,18 +520,15 @@ def kubectl_exec(
     pod_name = pods.items[0].metadata.name
 
     # Run command
-    attach_opts = ["-i", "-t"] if attach else []
     return utils.kubectl(
         "exec",
-        *attach_opts,
+        "--stdin",
+        "--tty",
         "--namespace",
         k8s_namespace(config),
         pod_name,
         "--",
-        "sh",
+        *command,
-        "-e",
-        "-c",
-        command,
     )
 
 
@@ -561,3 +580,4 @@ k8s.add_command(logs)
 k8s.add_command(wait)
 k8s.add_command(upgrade)
 k8s.add_command(apply_command)
+k8s.add_command(status)

tutor/templates/build/openedx/Dockerfile

@@ -33,7 +33,7 @@ RUN dockerize_url="https://github.com/powerman/dockerize/releases/download/$DOCK
 ###### Checkout edx-platform code
 FROM minimal as code
 ARG EDX_PLATFORM_REPOSITORY={{ EDX_PLATFORM_REPOSITORY }}
-ARG EDX_PLATFORM_VERSION={{ OPENEDX_COMMON_VERSION }}
+ARG EDX_PLATFORM_VERSION={{ EDX_PLATFORM_VERSION }}
 RUN mkdir -p /openedx/edx-platform && \
     git clone $EDX_PLATFORM_REPOSITORY --branch $EDX_PLATFORM_VERSION --depth 1 /openedx/edx-platform
 WORKDIR /openedx/edx-platform

tutor/templates/config/defaults.yml

@@ -19,6 +19,7 @@ DOCKER_IMAGE_PERMISSIONS: "{{ DOCKER_REGISTRY }}overhangio/openedx-permissions:{
 DOCKER_IMAGE_REDIS: "docker.io/redis:6.2.6"
 DOCKER_IMAGE_SMTP: "docker.io/devture/exim-relay:4.95-r0-2"
 EDX_PLATFORM_REPOSITORY: "https://github.com/openedx/edx-platform.git"
+EDX_PLATFORM_VERSION: "{{ OPENEDX_COMMON_VERSION }}"
 ELASTICSEARCH_HOST: "elasticsearch"
 ELASTICSEARCH_PORT: 9200
 ELASTICSEARCH_SCHEME: "http"

tutor/templates/hooks/mysql/init

@@ -15,5 +15,7 @@ done
 echo "MySQL is up and running"
 
 # edx-platform database
-mysql -u {{ MYSQL_ROOT_USERNAME }} --password="{{ MYSQL_ROOT_PASSWORD }}" --host "{{ MYSQL_HOST }}" --port {{ MYSQL_PORT }} -e 'CREATE DATABASE IF NOT EXISTS {{ OPENEDX_MYSQL_DATABASE }};'
+mysql -u {{ MYSQL_ROOT_USERNAME }} --password="{{ MYSQL_ROOT_PASSWORD }}" --host "{{ MYSQL_HOST }}" --port {{ MYSQL_PORT }} -e "CREATE DATABASE IF NOT EXISTS {{ OPENEDX_MYSQL_DATABASE }};"
-mysql -u {{ MYSQL_ROOT_USERNAME }} --password="{{ MYSQL_ROOT_PASSWORD }}" --host "{{ MYSQL_HOST }}" --port {{ MYSQL_PORT }} -e 'GRANT ALL ON {{ OPENEDX_MYSQL_DATABASE }}.* TO "{{ OPENEDX_MYSQL_USERNAME }}"@"%" IDENTIFIED BY "{{ OPENEDX_MYSQL_PASSWORD }}";'
+mysql -u {{ MYSQL_ROOT_USERNAME }} --password="{{ MYSQL_ROOT_PASSWORD }}" --host "{{ MYSQL_HOST }}" --port {{ MYSQL_PORT }} -e "CREATE USER IF NOT EXISTS '{{ OPENEDX_MYSQL_USERNAME }}';"
+mysql -u {{ MYSQL_ROOT_USERNAME }} --password="{{ MYSQL_ROOT_PASSWORD }}" --host "{{ MYSQL_HOST }}" --port {{ MYSQL_PORT }} -e "ALTER USER '{{ OPENEDX_MYSQL_USERNAME }}'@'%' IDENTIFIED BY '{{ OPENEDX_MYSQL_PASSWORD }}';"
+mysql -u {{ MYSQL_ROOT_USERNAME }} --password="{{ MYSQL_ROOT_PASSWORD }}" --host "{{ MYSQL_HOST }}" --port {{ MYSQL_PORT }} -e "GRANT ALL ON {{ OPENEDX_MYSQL_DATABASE }}.* TO '{{ OPENEDX_MYSQL_USERNAME }}'@'%';"