Towards a stable Kubernetes integration

Missing features:
- https certificates
- xqueue
- lms/cms workers

Moreover, we scalability issues due to the uploaded file storage in the
lms/cms. To address this issue we need to develop the MinIO plugin so
that it becomes compatible with Open edX.

Close #126 #179 #187

docs/k8s.rst

@@ -5,49 +5,78 @@ Kubernetes deployment
 
 With the same docker images we created for :ref:`single server deployment <local>` and :ref:`local development <development>`, we can launch an Open edX platform on Kubernetes. Always in 1 click, of course :)
 
-::
+A word of warning: managing a Kubernetes platform is a fairly advanced endeavour. In this documentation, we assume familiarity with Kubernetes. Running an Open edX platform with Tutor on a single server or in a Kubernetes cluster are two very different things. The local Open edX install was designed such that users with no prior experience with system administration could still launch an Open edX platform. It is *not* the case for the installation method outlined here. You have been warned :)
-
-           _       _              __            _                  
-      __ _| |_ __ | |__   __ _   / _| ___  __ _| |_ _   _ _ __ ___ 
-     / _` | | '_ \| '_ \ / _` | | |_ / _ \/ _` | __| | | | '__/ _ \
-    | (_| | | |_) | | | | (_| | |  _|  __/ (_| | |_| |_| | | |  __/
-     \__,_|_| .__/|_| |_|\__,_| |_|  \___|\__,_|\__|\__,_|_|  \___|
-            |_|                                                    
-
-Kubernetes deployment is currently an alpha feature, and we are hard at work to make it 100% reliable 🛠️ If you are interested in deploying Open edX to Kubernetes, please get in touch! Your input will be much appreciated.
 
 Requirements
 ------------
 
-In the following, we assume you have a working Kubernetes platform. For a start, you can run Kubernetes locally inside a VM with Minikube. Just follow the `official documentation <https://kubernetes.io/docs/setup/minikube/>`_.
+Memory
+~~~~~~
 
-Start Minikube::
+In the following, we assume you have access to a working Kubernetes cluster. `kubectl` should use your cluster configuration by default. To launch a cluster locally, you may try out Minikube. Just follow the `official installation instructions <https://kubernetes.io/docs/setup/minikube/>`_.
-
-    minikube start
-
-When minikube starts, it spawns a virtual machine (VM) which you can configure in your VM manager: on most platforms, this is Virtualbox. You should configure your VM to have at least 4Gb RAM; otherwise, database migrations will crash halfway, and that's a nasty issue...
 
+The Kubernetes cluster should have at least 4Gb of RAM on each node. When running Minikube, the virtual machine should have that much allocated memory. See below for an example with VirtualBox::
+  
 .. image:: img/virtualbox-minikube-system.png
     :alt: Virtualbox memory settings for Minikube
 
-Ingress addon must be installed::
+Ingress controller
+~~~~~~~~~~~~~~~~~~
 
+In order to access your platform, you will have to setup an Ingress controller. Instructions vary for each cloud provider. To deploy an Nginx Ingress controller, it might be as simple as running::
+
+    kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.24.1/deploy/mandatory.yaml
+    kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.24.1/deploy/provider/cloud-generic.yaml
+
+See the `official instructions <https://kubernetes.github.io/ingress-nginx/deploy/>`_ for more details.
+
+On Minikube, run::
+  
     minikube addons enable ingress
 
-At any point, access a UI to view the state of the platform with::
+With Kubernetes, your Open edX platform will *not* be available at localhost or studio.localhost. Instead, you will have to access your platform with the domain names you specified for the LMS and the CMS. To do so on a local computer, you will need to add the following line to /etc/hosts::
-
-    minikube dashboard
-
-With Kubernetes, your Open edX platform will not be available at localhost or studio.localhost. Instead, you will have to access your platform with the domain names you specified for the LMS and the CMS. To do so on a local computer, you will need to add the following line to /etc/hosts::
 
     MINIKUBEIP yourdomain.com studio.yourdomain.com preview.yourdomain.com notes.yourdomain.com
 
 where ``MINIKUBEIP`` should be replaced by the result of the command ``minikube ip``.
 
+`ReadWriteMany` storage provider access mode
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Some of the data volumes are shared between pods and thus require the `ReadWriteMany` access mode. We assume that a persistent volume provisioner with such capability is already installed on the cluster. For instance, on AWS the `AWS EBS <https://kubernetes.io/docs/concepts/storage/storage-classes/#aws-ebs>`_ provisioner is available. On DigitalOcean, there is `no such provider <https://www.digitalocean.com/docs/kubernetes/how-to/add-volumes/>`_ out of the box and you have to install one yourself.
+
+On Minikube, the standard storage class uses the `k8s.io/minikube-hostpath <https://kubernetes.io/docs/concepts/storage/volumes/#hostpath>`_ provider, which supports `ReadWriteMany` access mode out of the box, so there is no need to install an extra provider. 
+
+Kubernetes dashboard
+~~~~~~~~~~~~~~~~~~~~
+
+This is not a requirement per se, but it's very convenient to have a visual interface of the Kubernetes cluster. We suggest the official `Kubernetes dashboard <https://github.com/kubernetes/dashboard/>`_. Depending on your Kubernetes provider, you may need to install a dashboard yourself. There are generic instructions on the `project's README <https://github.com/kubernetes/dashboard/blob/master/README.md>`_. AWS provides `specific instructions <https://docs.aws.amazon.com/eks/latest/userguide/dashboard-tutorial.html>`_.
+
+On Minikube, the dashboard is already installed. To access the dashboard, run::
+  
+    minikube dashboard
+
+Technical details
+-----------------
+
+Under the hood, Tutor wraps ``kubectl`` commands to interact with the cluster. The various commands called by Tutor are printed in the console, so that you can reproduce and modify them yourself.
+
+Basically, the whole platform is described in manifest files stored in ``$(tutor config printroot)/env/k8s``. There is also a ``kustomization.yml`` file at the project root for `declarative application management <https://kubectl.docs.kubernetes.io/pages/app_management/apply.html>`_. This allows us to start and update resources with commands similar to ``kubectl apply -k $(tutor config printroot) --selector=...`` (see the ``kubectl apply`` `official documentation <https://kubectl.docs.kubernetes.io/pages/app_management/apply.html>`_).
+
+The other benefit of ``kubectl apply`` is that it allows you to customise the Kubernetes resources as much as you want. For instance, the default Tutor configuration can be extended by a ``kustomization.yml`` file stored in ``$(tutor config printroot)/env-custom/`` and which would start with::
+  
+    apiVersion: kustomize.config.k8s.io/v1beta1
+    kind: Kustomization
+    bases:
+    - ../env/
+    ...
+
+To learn more about "kustomizations", refer to the `official documentation <https://kubectl.docs.kubernetes.io/pages/app_customization/introduction.html>`_.
+
 Quickstart
 ----------
 
-Launch the platform on k8s in 1 click::
+Launch the platform on Kubernetes in one command::
 
     tutor k8s quickstart
 
@@ -56,34 +85,14 @@ All Kubernetes resources are associated to the "openedx" namespace. If you don't
 .. image:: img/k8s-dashboard.png
     :alt: Kubernetes dashboard ("openedx" namespace)
 
-Upgrading
+The same ``tutor k8s quickstart`` command can be used to upgrade the cluster to the latest version.
----------
 
-After pulling updates from the Tutor repository, you can apply changes with::
+Other commands
+--------------
 
-    tutor k8s stop
+As with the :ref:`local installation <local>`, there are multiple commands to run operations on your Open edX platform. To view those commands, run::
-    tutor k8s start
+  
-
+    tutor k8s -h
-Accessing the Kubernetes dashboard
-----------------------------------
-
-Depending on your Kubernetes provider, you may need to create a dashboard yourself. To do so, run::
-
-	kubectl create -f https://raw.githubusercontent.com/kubernetes/dashboard/master/aio/deploy/recommended/kubernetes-dashboard.yaml
-
-Then, you will have to create an admin user::
-
-    tutor k8s adminuser
-
-Print the admin token required for authentication, and copy its value::
-
-    tutor k8s admintoken
-
-Create a proxy to the Kubernetes API server::
-
-	kubectl proxy
-
-Use the token to log in the dashboard at the following url: http://localhost:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/
 
 Missing features
 ----------------
@@ -92,6 +101,5 @@ For now, the following features from the local deployment are not supported:
 
 - HTTPS certificates
 - Xqueue
-- Student notes
 
 Kubernetes deployment is under intense development, and these features should be implemented pretty soon. Stay tuned 🤓

plugins/minio/tutorminio/patches/k8s-jobs.patch

@@ -0,0 +1,22 @@
+---
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: minio-client/init
+  labels:
+    app.kubernetes.io/component: script
+    app.kubernetes.io/name: minio-client/init
+spec:
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: minio-client/init
+    spec:
+      restartPolicy: Never
+      containers:
+      - name: minio-client
+        image: {{ MINIO_DOCKER_REGISTRY }}{{ MINIO_DOCKER_IMAGE_CLIENT }}
+        command:  ["/bin/sh", "-e", "-c"]
+        args: 
+          - |
+            {{ include_plugin_script("minio", "minio-client", "init")|indent(12) }}

requirements/base.in

@@ -2,5 +2,6 @@ appdirs
 click>=7.0
 click_repl
 jinja2
+# TODO get rid of kubernetes?
 kubernetes
 pyyaml>=4.2b1

tests/test_config.py

@@ -32,7 +32,7 @@ class ConfigTests(unittest.TestCase):
         self.assertEqual("abcd", config["MYSQL_ROOT_PASSWORD"])
 
     @unittest.mock.patch.object(tutor_config.fmt, "echo")
-    def test_save_twice(self, mock_echo):
+    def test_save_twice(self, _):
         with tempfile.TemporaryDirectory() as root:
             tutor_config.save(root, silent=True)
             config1 = tutor_config.load_user(root)
@@ -43,20 +43,20 @@ class ConfigTests(unittest.TestCase):
         self.assertEqual(config1, config2)
 
     @unittest.mock.patch.object(tutor_config.fmt, "echo")
-    def test_removed_entry_is_added_on_save(self, mock_echo):
+    def test_removed_entry_is_added_on_save(self, _):
         with tempfile.TemporaryDirectory() as root:
             with unittest.mock.patch.object(
                 tutor_config.utils, "random_string"
             ) as mock_random_string:
                 mock_random_string.return_value = "abcd"
-                config1, defaults = tutor_config.load_all(root)
+                config1, _ = tutor_config.load_all(root)
                 password1 = config1["MYSQL_ROOT_PASSWORD"]
 
                 config1.pop("MYSQL_ROOT_PASSWORD")
                 tutor_config.save_config(root, config1)
 
                 mock_random_string.return_value = "efgh"
-                config2, defaults = tutor_config.load_all(root)
+                config2, _ = tutor_config.load_all(root)
                 password2 = config2["MYSQL_ROOT_PASSWORD"]
 
         self.assertEqual("abcd", password1)

tutor/commands/k8s.py

@@ -2,7 +2,8 @@ import click
 
 from . import config as tutor_config
 from .. import env as tutor_env
-from .. import exceptions
+
+# from .. import exceptions
 from .. import fmt
 from .. import opts
 from .. import scripts
@@ -16,89 +17,80 @@ def k8s():
 
 @click.command(help="Configure and run Open edX from scratch")
 @opts.root
-def quickstart(root):
+@click.option("-y", "--yes", "silent", is_flag=True, help="Run non-interactively")
+def quickstart(root, silent):
     click.echo(fmt.title("Interactive platform configuration"))
-    tutor_config.save(root)
+    tutor_config.save(root, silent=silent)
-    click.echo(fmt.title("Stopping any existing platform"))
-    stop.callback()
     click.echo(fmt.title("Starting the platform"))
     start.callback(root)
-    click.echo(
+    click.echo(fmt.title("Database creation and migrations"))
-        fmt.title(
-            "Running migrations. NOTE: this might fail. If it does, please retry 'tutor k8s databases' later"
-        )
-    )
     databases.callback(root)
+    # TODO https certificates
 
 
 @click.command(help="Run all configured Open edX services")
 @opts.root
 def start(root):
-    config = tutor_config.load(root)
+    # Create namespace
-    kubectl_no_fail("create", "-f", tutor_env.pathjoin(root, "k8s", "namespace.yml"))
+    utils.kubectl(
-
+        "apply",
-    kubectl(
+        "--kustomize",
-        "create",
+        tutor_env.pathjoin(root),
-        "configmap",
+        "--wait",
-        "nginx-config",
+        "--selector",
-        "--from-file",
+        "app.kubernetes.io/component=namespace",
-        tutor_env.pathjoin(root, "apps", "nginx"),
     )
-    if config["ACTIVATE_MYSQL"]:
+    # Create volumes
-        kubectl(
+    utils.kubectl(
-            "create",
+        "apply",
-            "configmap",
+        "--kustomize",
-            "mysql-config",
+        tutor_env.pathjoin(root),
-            "--from-env-file",
+        "--wait",
-            tutor_env.pathjoin(root, "apps", "mysql", "auth.env"),
+        "--selector",
-        )
+        "app.kubernetes.io/component=volume",
-    kubectl(
-        "create",
-        "configmap",
-        "openedx-settings-lms",
-        "--from-file",
-        tutor_env.pathjoin(root, "apps", "openedx", "settings", "lms"),
     )
-    kubectl(
+    # Create everything else (except Job objects)
-        "create",
+    utils.kubectl(
-        "configmap",
+        "apply",
-        "openedx-settings-cms",
+        "--selector",
-        "--from-file",
+        "app.kubernetes.io/component!=script",
-        tutor_env.pathjoin(root, "apps", "openedx", "settings", "cms"),
+        "--kustomize",
+        tutor_env.pathjoin(root),
     )
-    kubectl(
-        "create",
-        "configmap",
-        "openedx-config",
-        "--from-file",
-        tutor_env.pathjoin(root, "apps", "openedx", "config"),
-    )
-
-    kubectl("create", "-f", tutor_env.pathjoin(root, "k8s", "volumes.yml"))
-    kubectl("create", "-f", tutor_env.pathjoin(root, "k8s", "ingress.yml"))
-    kubectl("create", "-f", tutor_env.pathjoin(root, "k8s", "services.yml"))
-    kubectl("create", "-f", tutor_env.pathjoin(root, "k8s", "deployments.yml"))
 
 
 @click.command(help="Stop a running platform")
-def stop():
+@opts.root
-    kubectl("delete", "deployments,services,ingress,configmaps", "--all")
+def stop(root):
+    config = tutor_config.load(root)
+    utils.kubectl(
+        "delete",
+        "--namespace",
+        config["K8S_NAMESPACE"],
+        "--selector=app.kubernetes.io/instance=openedx-" + config["ID"],
+        "deployments,services,ingress,configmaps,jobs",
+    )
 
 
 @click.command(help="Completely delete an existing platform")
+@opts.root
 @click.option("-y", "--yes", is_flag=True, help="Do not ask for confirmation")
-def delete(yes):
+def delete(root, yes):
     if not yes:
         click.confirm(
             "Are you sure you want to delete the platform? All data will be removed.",
             abort=True,
         )
-    kubectl("delete", "namespace", K8s.NAMESPACE)
+    config = tutor_config.load(root)
+    utils.kubectl(
+        "delete", "-k", tutor_env.pathjoin(root), "--ignore-not-found=true", "--wait"
+    )
 
 
 @click.command(help="Create databases and run database migrations")
 @opts.root
 def databases(root):
+    # TODO this requires a running mysql/mongodb/elasticsearch. Maybe we should wait until they are up?
     config = tutor_config.load(root)
     runner = K8sScriptRunner(root, config)
     scripts.migrate(runner)
@@ -113,6 +105,7 @@ def databases(root):
 def createuser(root, superuser, staff, name, email):
     config = tutor_config.load(root)
     runner = K8sScriptRunner(root, config)
+    # TODO this is not going to work
     scripts.create_user(runner, superuser, staff, name, email)
 
 
@@ -130,107 +123,70 @@ def importdemocourse(root):
 @click.command(help="Re-index courses for better searching")
 @opts.root
 def indexcourses(root):
-    # Note: this is currently broken with "pymongo.errors.ConnectionFailure: [Errno 111] Connection refused"
-    # I'm not quite sure the settings are correctly picked up. Which is weird because migrations work very well.
     config = tutor_config.load(root)
     runner = K8sScriptRunner(root, config)
+    # TODO this is not going to work
     scripts.index_courses(runner)
 
 
-@click.command(help="Launch a shell in LMS or CMS")
+# @click.command(help="Launch a shell in LMS or CMS")
-@click.argument("service", type=click.Choice(["lms", "cms"]))
+# @click.argument("service", type=click.Choice(["lms", "cms"]))
-def shell(service):
+# def shell(service):
-    K8s().execute(service, "bash")
+#     K8s().execute(service, "bash")
 
 
-@click.command(help="Create a Kubernetesadmin user")
+@click.command(help="View output from containers")
 @opts.root
-def adminuser(root):
+@click.option("-f", "--follow", is_flag=True, help="Follow log output")
-    utils.kubectl("create", "-f", tutor_env.pathjoin(root, "k8s", "adminuser.yml"))
+@click.option("--tail", type=int, help="Number of lines to show from each container")
+@click.argument("service")
+def logs(root, follow, tail, service):
+    config = tutor_config.load(root)
 
+    command = ["logs"]
+    command += ["--namespace", config["K8S_NAMESPACE"]]
 
-@click.command(help="Print the Kubernetes admin user token")
+    if follow:
-def admintoken():
+        command += ["--follow"]
-    click.echo(K8s().admin_token())
+    if tail is not None:
+        command += ["--tail", str(tail)]
 
+    selector = "--selector=app.kubernetes.io/instance=openedx-" + config["ID"]
+    if service:
+        selector += ",app.kubernetes.io/name=" + service
+    command.append(selector)
 
-def kubectl(*command):
+    utils.kubectl(*command)
-    """
-    Run kubectl commands in the right namespace. Also, errors are completely
-    ignored, to avoid stopping on "AlreadyExists" errors.
-    """
-    args = list(command)
-    args += ["--namespace", K8s.NAMESPACE]
-    kubectl_no_fail(*args)
-
-
-def kubectl_no_fail(*command):
-    """
-    Run kubectl commands and ignore exceptions, to avoid stopping on
-    "AlreadyExists" errors.
-    """
-    try:
-        utils.kubectl(*command)
-    except exceptions.TutorError:
-        pass
-
-
-class K8s:
-    CLIENT = None
-    NAMESPACE = "openedx"
-
-    def __init__(self):
-        pass
-
-    @property
-    def client(self):
-        if self.CLIENT is None:
-            # Import moved here for performance reasons
-            import kubernetes
-
-            kubernetes.config.load_kube_config()
-            self.CLIENT = kubernetes.client.CoreV1Api()
-        return self.CLIENT
-
-    def pod_name(self, app):
-        selector = "app=" + app
-        try:
-            return (
-                self.client.list_namespaced_pod("openedx", label_selector=selector)
-                .items[0]
-                .metadata.name
-            )
-        except IndexError:
-            raise exceptions.TutorError(
-                "Pod with app {} does not exist. Make sure that the pod is running."
-            )
-
-    def admin_token(self):
-        # Note: this is a HORRIBLE way of looking for a secret
-        try:
-            secret = [
-                s
-                for s in self.client.list_namespaced_secret("kube-system").items
-                if s.metadata.name.startswith("admin-user-token")
-            ][0]
-        except IndexError:
-            raise exceptions.TutorError(
-                "Secret 'admin-user-token'. Make sure that admin user was created."
-            )
-        return self.client.read_namespaced_secret(
-            secret.metadata.name, "kube-system"
-        ).data["token"]
-
-    def execute(self, app, *command):
-        podname = self.pod_name(app)
-        kubectl_no_fail(
-            "exec", "--namespace", self.NAMESPACE, "-it", podname, "--", *command
-        )
 
 
 class K8sScriptRunner(scripts.BaseRunner):
-    def exec(self, service, command):
+    def run(self, service, script, config=None):
-        K8s().execute(service, "sh", "-e", "-c", command)
+        job_name = "{}/{}".format(service, script)
+        selector = (
+            "--selector=app.kubernetes.io/component=script,app.kubernetes.io/name="
+            + job_name
+        )
+        kustomization = tutor_env.pathjoin(self.root)
+        # Delete any previously run jobs (completed job objects still exist)
+        utils.kubectl("delete", "-k", kustomization, "--wait", selector)
+        # Run job
+        utils.kubectl("apply", "-k", kustomization, selector)
+        # Wait until complete
+        fmt.echo_info(
+            "Waiting for job to complete. To view logs, run: \n\n    kubectl logs -n {} -l app.kubernetes.io/name={} --follow\n".format(
+                self.config["K8S_NAMESPACE"], job_name
+            )
+        )
+        utils.kubectl(
+            "wait",
+            "--namespace",
+            self.config["K8S_NAMESPACE"],
+            "--for=condition=complete",
+            "--timeout=-1s",
+            selector,
+            "job",
+        )
+        # TODO check failure?
 
 
 k8s.add_command(quickstart)
@@ -241,6 +197,5 @@ k8s.add_command(databases)
 k8s.add_command(createuser)
 k8s.add_command(importdemocourse)
 k8s.add_command(indexcourses)
-k8s.add_command(shell)
+# k8s.add_command(shell)
-k8s.add_command(adminuser)
+k8s.add_command(logs)
-k8s.add_command(admintoken)

tutor/env.py

@@ -28,6 +28,7 @@ class Renderer:
             environment.filters["random_string"] = utils.random_string
             environment.filters["common_domain"] = utils.common_domain
             environment.filters["reverse_host"] = utils.reverse_host
+            environment.filters["walk_templates"] = walk_templates
             environment.globals["TUTOR_VERSION"] = __version__
             cls.ENVIRONMENT = environment
 
@@ -97,6 +98,7 @@ def render_full(root, config):
         save_subdir(subdir, root, config)
     copy_subdir("build", root)
     save_file(VERSION_FILENAME, root, config)
+    save_file("kustomization.yml", root, config)
 
 
 def save_subdir(subdir, root, config):

tutor/templates/config.yml

@@ -50,6 +50,7 @@ LOCAL_PROJECT_NAME: "tutor_local"
 ELASTICSEARCH_HOST: "elasticsearch"
 ELASTICSEARCH_PORT: 9200
 FORUM_HOST: "forum"
+K8S_NAMESPACE: "openedx"
 LANGUAGE_CODE: "en"
 MEMCACHED_HOST: "memcached"
 MEMCACHED_PORT: 11211

tutor/templates/k8s/adminuser.yml

@@ -1,20 +0,0 @@
----
-apiVersion: v1
-kind: ServiceAccount
-metadata:
-  name: admin-user
-  namespace: kube-system
-
----
-apiVersion: rbac.authorization.k8s.io/v1
-kind: ClusterRoleBinding
-metadata:
-  name: admin-user
-roleRef:
-  apiGroup: rbac.authorization.k8s.io
-  kind: ClusterRole
-  name: cluster-admin
-subjects:
-- kind: ServiceAccount
-  name: admin-user
-  namespace: kube-system

tutor/templates/k8s/deployments.yml

@@ -3,15 +3,16 @@ apiVersion: apps/v1
 kind: Deployment
 metadata:
   name: cms
+  labels:
+    app.kubernetes.io/name: cms
 spec:
-  replicas: 1
   selector:
     matchLabels:
-      app: cms
+      app.kubernetes.io/name: cms
   template:
     metadata:
       labels:
-        app: cms
+        app.kubernetes.io/name: cms
     spec:
       containers:
         - name: cms
@@ -30,7 +31,6 @@ spec:
               name: config
             - mountPath: /openedx/data
               name: data
-          #imagePullPolicy: Always
       volumes:
         - name: settings-lms
           configMap:
@@ -50,22 +50,22 @@ apiVersion: apps/v1
 kind: Deployment
 metadata:
   name: forum
+  labels:
+    app.kubernetes.io/name: forum
 spec:
-  replicas: 1
   selector:
     matchLabels:
-      app: forum
+      app.kubernetes.io/name: forum
   template:
     metadata:
       labels:
-        app: forum
+        app.kubernetes.io/name: forum
     spec:
       containers:
         - name: forum
           image: {{ DOCKER_REGISTRY }}{{ DOCKER_IMAGE_FORUM }}
           ports:
             - containerPort: 4567
-          imagePullPolicy: Always
           env:
             - name: SEARCH_SERVER
               value: "http://{{ ELASTICSEARCH_HOST }}:{{ ELASTICSEARCH_PORT }}"
@@ -81,15 +81,16 @@ apiVersion: apps/v1
 kind: Deployment
 metadata:
   name: lms
+  labels:
+    app.kubernetes.io/name: lms
 spec:
-  replicas: 1
   selector:
     matchLabels:
-      app: lms
+      app.kubernetes.io/name: lms
   template:
     metadata:
       labels:
-        app: lms
+        app.kubernetes.io/name: lms
     spec:
       containers:
         - name: lms
@@ -105,7 +106,6 @@ spec:
               name: config
             - mountPath: /openedx/data
               name: data
-          imagePullPolicy: Always
       volumes:
         - name: settings-lms
           configMap:
@@ -125,15 +125,16 @@ apiVersion: apps/v1
 kind: Deployment
 metadata:
   name: elasticsearch
+  labels:
+    app.kubernetes.io/name: elasticearch
 spec:
-  replicas: 1
   selector:
     matchLabels:
-      app: elasticsearch
+      app.kubernetes.io/name: elasticsearch
   template:
     metadata:
       labels:
-        app: elasticsearch
+        app.kubernetes.io/name: elasticsearch
     spec:
       containers:
         - name: elasticsearch
@@ -161,15 +162,16 @@ apiVersion: apps/v1
 kind: Deployment
 metadata:
   name: memcached
+  labels:
+    app.kubernetes.io/name: memcached
 spec:
-  replicas: 1
   selector:
     matchLabels:
-      app: memcached
+      app.kubernetes.io/name: memcached
   template:
     metadata:
       labels:
-        app: memcached
+        app.kubernetes.io/name: memcached
     spec:
       containers:
         - name: memcached
@@ -183,20 +185,21 @@ apiVersion: apps/v1
 kind: Deployment
 metadata:
   name: mongodb
+  labels:
+    app.kubernetes.io/name: mongodb
 spec:
-  replicas: 1
   selector:
     matchLabels:
-      app: mongodb
+      app.kubernetes.io/name: mongodb
   template:
     metadata:
       labels:
-        app: mongodb
+        app.kubernetes.io/name: mongodb
     spec:
       containers:
         - name: mongodb
           image: {{ DOCKER_REGISTRY }}{{ DOCKER_IMAGE_MONGODB }}
-          command: ["mongod", "--smallfiles", "--nojournal", "--storageEngine", "wiredTiger"]
+          args: ["mongod", "--smallfiles", "--nojournal", "--storageEngine", "wiredTiger"]
           ports:
             - containerPort: 27017
           volumeMounts:
@@ -204,6 +207,7 @@ spec:
               name: data
       volumes:
         - name: data
+          # TODO this should be a pvc, otherwise the volume data will be lost when the pod is deleted
           emptyDir: {}
 {% endif %}
 {% if ACTIVATE_MYSQL %}
@@ -212,19 +216,21 @@ apiVersion: apps/v1
 kind: Deployment
 metadata:
   name: mysql
+  labels:
+    app.kubernetes.io/name: mysql
 spec:
-  replicas: 1
   selector:
     matchLabels:
-      app: mysql
+      app.kubernetes.io/name: mysql
   template:
     metadata:
       labels:
-        app: mysql
+        app.kubernetes.io/name: mysql
     spec:
       containers:
         - name: mysql
           image: {{ DOCKER_REGISTRY }}{{ DOCKER_IMAGE_MYSQL }}
+          args: ["mysqld", "--character-set-server=utf8", "--collation-server=utf8_general_ci"]
           env:
             - name: MYSQL_ROOT_PASSWORD
               valueFrom:
@@ -241,21 +247,60 @@ spec:
           persistentVolumeClaim:
             claimName: mysql
 {% endif %}
+{% if ACTIVATE_NOTES %}
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: notes
+  labels:
+    app.kubernetes.io/name: notes
+spec:
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: notes
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: notes
+    spec:
+      containers:
+        - name: notes
+          image: {{ DOCKER_REGISTRY }}{{ DOCKER_IMAGE_NOTES }}
+          ports:
+            - containerPort: 8000
+          env:
+            - name: DJANGO_SETTINGS_MODULE
+              value: notesserver.settings.tutor
+          volumeMounts:
+            - mountPath: /openedx/edx-notes-api/notesserver/settings/tutor.py
+              name: settings
+            - mountPath: /openedx/data
+              name: data
+      volumes:
+        - name: data
+          persistentVolumeClaim:
+            claimName: notes-data
+        - name: settings
+          configMap:
+            name: notes-settings
+{% endif %}
 {% if ACTIVATE_SMTP %}
 ---
 apiVersion: apps/v1
 kind: Deployment
 metadata:
   name: smtp
+  labels:
+    app.kubernetes.io/name: smtp
 spec:
-  replicas: 1
   selector:
     matchLabels:
-      app: smtp
+      app.kubernetes.io/name: smtp
   template:
     metadata:
       labels:
-        app: smtp
+        app.kubernetes.io/name: smtp
     spec:
       containers:
         - name: smtp
@@ -268,31 +313,30 @@ apiVersion: apps/v1
 kind: Deployment
 metadata:
   name: nginx
+  labels:
+    app.kubernetes.io/name: nginx
 spec:
-  replicas: 1
   selector:
     matchLabels:
-      app: nginx
+      app.kubernetes.io/name: nginx
   template:
     metadata:
       labels:
-        app: nginx
+        app.kubernetes.io/name: nginx
     spec:
       initContainers:
         - name: clean-openedx-staticfiles
           image: {{ DOCKER_REGISTRY }}{{ DOCKER_IMAGE_OPENEDX }}
-          command: ['rm', '-rf', '/var/www/openedx/staticfiles']
+          args: ['rm', '-rf', '/var/www/openedx/staticfiles']
           volumeMounts:
             - mountPath: /var/www/openedx/
               name: openedx-staticfiles
-          imagePullPolicy: Always
         - name: init-openedx-staticfiles
           image: {{ DOCKER_REGISTRY }}{{ DOCKER_IMAGE_OPENEDX }}
-          command: ['cp', '-r', '/openedx/staticfiles', '/var/www/openedx/']
+          args: ['cp', '-r', '/openedx/staticfiles', '/var/www/openedx/']
           volumeMounts:
             - mountPath: /var/www/openedx/
               name: openedx-staticfiles
-          imagePullPolicy: Always
       containers:
         - name: nginx
           image: {{ DOCKER_REGISTRY }}{{ DOCKER_IMAGE_NGINX }}
@@ -301,38 +345,46 @@ spec:
               name: config
             - mountPath: /var/www/openedx/
               name: openedx-staticfiles
+              readOnly: true
+            - mountPath: /openedx/data/cms
+              name: data-cms
+              readOnly: true
             - mountPath: /openedx/data/lms
-              name: data
+              name: data-lms
+              readOnly: true
           ports:
             - containerPort: 80
-              name: http-port
             - containerPort: 443
-              name: https-port
       volumes:
         - name: config
           configMap:
             name: nginx-config
         - name: openedx-staticfiles
+          emptyDir: {}
+        - name: data-cms
           persistentVolumeClaim:
-            claimName: openedx-staticfiles
+            claimName: cms-data
-        - name: data
+            readOnly: true
+        - name: data-lms
           persistentVolumeClaim:
             claimName: lms-data
+            readOnly: true
 {% if ACTIVATE_RABBITMQ %}
 ---
 apiVersion: apps/v1
 kind: Deployment
 metadata:
   name: rabbitmq
+  labels:
+    app.kubernetes.io/name: rabbitmq
 spec:
-  replicas: 1
   selector:
     matchLabels:
-      app: rabbitmq
+      app.kubernetes.io/name: rabbitmq
   template:
     metadata:
       labels:
-        app: rabbitmq
+        app.kubernetes.io/name: rabbitmq
     spec:
       containers:
         - name: rabbitmq

tutor/templates/k8s/ingress.yml

@@ -5,9 +5,7 @@ metadata:
   name: web
 spec:
   rules:
-  {% set hosts = [LMS_HOST, "preview." + LMS_HOST, CMS_HOST] %}
+  {% set hosts = [LMS_HOST, "preview." + LMS_HOST, CMS_HOST] %}{% if ACTIVATE_NOTES %}{% set hosts = hosts + [NOTES_HOST] %}{% endif %}{% for host in hosts %}
-  {% if ACTIVATE_NOTES %}{% set hosts = hosts + [NOTES_HOST] %}{% endif %}
-  {% for host in hosts %}
   - host: {{ host }}
     http:
       paths:

tutor/templates/k8s/jobs.yml

@@ -0,0 +1,237 @@
+{% macro include_script(script) %}{% include "scripts/" + script %}{% endmacro %}---
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: mysql-client/init
+  labels:
+    app.kubernetes.io/component: script
+    app.kubernetes.io/name: mysql-client/init
+spec:
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: mysql-client/init
+    spec:
+      restartPolicy: Never
+      containers:
+      - name: mysql-client
+        image: {{ DOCKER_REGISTRY }}{{ DOCKER_IMAGE_MYSQL }}
+        command:  ["/bin/sh", "-e", "-c"]
+        args: 
+          - |
+            {{ include_script("mysql-client/createdatabases")|indent(12) }}
+---
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: cms/init
+  labels:
+    app.kubernetes.io/component: script
+    app.kubernetes.io/name: cms/init
+spec:
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: cms/init
+    spec:
+      restartPolicy: Never
+      containers:
+      - name: cms
+        image: {{ DOCKER_REGISTRY }}{{ DOCKER_IMAGE_OPENEDX }}
+        command:  ["/bin/sh", "-e", "-c"]
+        args: 
+          - |
+            {{ include_script("cms/init")|indent(12) }}
+        volumeMounts:
+          - mountPath: /openedx/edx-platform/lms/envs/tutor/
+            name: settings-lms
+          - mountPath: /openedx/edx-platform/cms/envs/tutor/
+            name: settings-cms
+          - mountPath: /openedx/config
+            name: config
+      volumes:
+      - name: settings-lms
+        configMap:
+          name: openedx-settings-lms
+      - name: settings-cms
+        configMap:
+          name: openedx-settings-cms
+      - name: config
+        configMap:
+          name: openedx-config
+---
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: lms/init
+  labels:
+    app.kubernetes.io/component: script
+    app.kubernetes.io/name: lms/init
+spec:
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: lms/init
+    spec:
+      restartPolicy: Never
+      containers:
+      - name: lms
+        image: {{ DOCKER_REGISTRY }}{{ DOCKER_IMAGE_OPENEDX }}
+        command:  ["/bin/sh", "-e", "-c"]
+        args: 
+          - |
+            {{ include_script("lms/init")|indent(12) }}
+        volumeMounts:
+          - mountPath: /openedx/edx-platform/lms/envs/tutor/
+            name: settings-lms
+          - mountPath: /openedx/edx-platform/cms/envs/tutor/
+            name: settings-cms
+          - mountPath: /openedx/config
+            name: config
+      volumes:
+      - name: settings-lms
+        configMap:
+          name: openedx-settings-lms
+      - name: settings-cms
+        configMap:
+          name: openedx-settings-cms
+      - name: config
+        configMap:
+          name: openedx-config
+---
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: cms/importdemocourse
+  labels:
+    app.kubernetes.io/component: script
+    app.kubernetes.io/name: cms/importdemocourse
+spec:
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: cms/importdemocourse
+    spec:
+      restartPolicy: Never
+      containers:
+      - name: lms
+        image: {{ DOCKER_REGISTRY }}{{ DOCKER_IMAGE_OPENEDX }}
+        command:  ["/bin/sh", "-e", "-c"]
+        args: 
+          - |
+            {{ include_script("cms/importdemocourse")|indent(12) }}
+        volumeMounts:
+          - mountPath: /openedx/edx-platform/lms/envs/tutor/
+            name: settings-lms
+          - mountPath: /openedx/edx-platform/cms/envs/tutor/
+            name: settings-cms
+          - mountPath: /openedx/config
+            name: config
+          - mountPath: /openedx/data
+            name: data
+      volumes:
+      - name: settings-lms
+        configMap:
+          name: openedx-settings-lms
+      - name: settings-cms
+        configMap:
+          name: openedx-settings-cms
+      - name: config
+        configMap:
+          name: openedx-config
+      - name: data
+        persistentVolumeClaim:
+          claimName: lms-data
+{% if ACTIVATE_FORUM %}
+---
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: forum/init
+  labels:
+    app.kubernetes.io/component: script
+    app.kubernetes.io/name: forum/init
+spec:
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: forum/init
+    spec:
+      restartPolicy: Never
+      containers:
+      - name: forum
+        image: {{ DOCKER_REGISTRY }}{{ DOCKER_IMAGE_FORUM }}
+        command:  ["/bin/sh", "-e", "-c"]
+        args: 
+          - |
+            {{ include_script("forum/init")|indent(12) }}
+        env:
+          - name: SEARCH_SERVER
+            value: "http://{{ ELASTICSEARCH_HOST }}:{{ ELASTICSEARCH_PORT }}"
+          - name: MONGOHQ_URL
+            value: "mongodb://{% if MONGODB_USERNAME and MONGODB_PASSWORD %}{{ MONGODB_USERNAME}}:{{ MONGODB_PASSWORD }}@{% endif %}{{ MONGODB_HOST }}:{{ MONGODB_PORT }}/cs_comments_service"
+{% endif %}
+{% if ACTIVATE_NOTES %}
+---
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: notes/init
+  labels:
+    app.kubernetes.io/component: script
+    app.kubernetes.io/name: notes/init
+spec:
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: notes/init
+    spec:
+      restartPolicy: Never
+      containers:
+      - name: notes
+        image: {{ DOCKER_REGISTRY }}{{ DOCKER_IMAGE_NOTES }}
+        command:  ["/bin/sh", "-e", "-c"]
+        args: 
+          - |
+            {{ include_script("notes/init")|indent(12) }}
+        volumeMounts:
+          - mountPath: /openedx/edx-notes-api/notesserver/settings/tutor.py
+            name: settings
+      volumes:
+        - name: settings
+          configMap:
+            name: notes-settings
+{% endif %}
+{% if ACTIVATE_XQUEUE %}
+---
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: xqueue/init
+  labels:
+    app.kubernetes.io/component: script
+    app.kubernetes.io/name: xqueue/init
+spec:
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: xqueue/init
+    spec:
+      restartPolicy: Never
+      containers:
+      - name: xqueue
+        image: {{ DOCKER_REGISTRY }}{{ DOCKER_IMAGE_NOTES }}
+        command:  ["/bin/sh", "-e", "-c"]
+        args: 
+          - |
+            {{ include_script("xqueue/init")|indent(12) }}
+        volumeMounts:
+          - mountPath: /openedx/xqueue/xqueue/tutor.py
+            name: settings
+      volumes:
+        - name: settings
+          configMap:
+            name: notes-settings
+{% endif %}
+
+{{ patch("k8s-jobs") }}

tutor/templates/k8s/namespace.yml

@@ -2,4 +2,6 @@
 apiVersion: v1
 kind: Namespace
 metadata:
-  name: openedx
+  name: {{ K8S_NAMESPACE }}
+  labels:
+    app.kubernetes.io/component: namespace

tutor/templates/k8s/services.yml

@@ -9,7 +9,7 @@ spec:
     - port: 8000
       protocol: TCP
   selector:
-    app: cms
+    app.kubernetes.io/name: cms
 {% if ACTIVATE_FORUM %}
 ---
 apiVersion: v1
@@ -22,7 +22,7 @@ spec:
     - port: 4567
       protocol: TCP
   selector:
-    app: forum
+    app.kubernetes.io/name: forum
 {% endif %}
 ---
 apiVersion: v1
@@ -35,7 +35,7 @@ spec:
     - port: 8000
       protocol: TCP
   selector:
-    app: lms
+    app.kubernetes.io/name: lms
 {% if ACTIVATE_ELASTICSEARCH %}
 ---
 apiVersion: v1
@@ -48,7 +48,7 @@ spec:
     - port: 9200
       protocol: TCP
   selector:
-    app: elasticsearch
+    app.kubernetes.io/name: elasticsearch
 {% endif %}
 {% if ACTIVATE_MEMCACHED %}
 ---
@@ -62,7 +62,7 @@ spec:
     - port: 11211
       protocol: TCP
   selector:
-    app: memcached
+    app.kubernetes.io/name: memcached
 {% endif %}
 {% if ACTIVATE_MONGODB %}
 ---
@@ -76,7 +76,7 @@ spec:
     - port: 27017
       protocol: TCP
   selector:
-    app: mongodb
+    app.kubernetes.io/name: mongodb
 {% endif %}
 {% if ACTIVATE_MYSQL %}
 ---
@@ -90,7 +90,7 @@ spec:
     - port: 3306
       protocol: TCP
   selector:
-    app: mysql
+    app.kubernetes.io/name: mysql
 {% endif %}
 ---
 apiVersion: v1
@@ -101,15 +101,25 @@ spec:
   type: NodePort
   ports:
     - port: 80
-      protocol: TCP
       name: http
-      targetPort: http-port
     - port: 443
-      protocol: TCP
       name: https
-      targetPort: https-port
   selector:
-    app: nginx
+    app.kubernetes.io/name: nginx
+{% if ACTIVATE_FORUM %}
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: notes
+spec:
+  type: NodePort
+  ports:
+    - port: 8000
+      protocol: TCP
+  selector:
+    app.kubernetes.io/name: notes
+{% endif %}
 {% if ACTIVATE_RABBITMQ %}
 ---
 apiVersion: v1
@@ -122,7 +132,7 @@ spec:
     - port: 5672
       protocol: TCP
   selector:
-    app: rabbitmq
+    app.kubernetes.io/name: rabbitmq
 {% endif %}
 {% if ACTIVATE_SMTP %}
 ---
@@ -136,5 +146,5 @@ spec:
     - port: 25
       protocol: TCP
   selector:
-    app: smtp
+    app.kubernetes.io/name: smtp
 {% endif %}

tutor/templates/k8s/volumes.yml

@@ -3,21 +3,26 @@ apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:
   name: cms-data
+  labels:
+    app.kubernetes.io/component: volume
+    app.kubernetes.io/name: cms-data
 spec:
   accessModes:
-    - ReadWriteOnce
+    - ReadWriteMany
   resources:
     requests:
       storage: 2Gi
-
 ---
 apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:
   name: lms-data
+  labels:
+    app.kubernetes.io/component: volume
+    app.kubernetes.io/name: lms-data
 spec:
   accessModes:
-    - ReadWriteOnce
+    - ReadWriteMany
   resources:
     requests:
       storage: 2Gi
@@ -27,6 +32,9 @@ apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:
   name: elasticsearch
+  labels:
+    app.kubernetes.io/component: volume
+    app.kubernetes.io/name: elasticearch
 spec:
   accessModes:
     - ReadWriteOnce
@@ -40,6 +48,9 @@ apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:
   name: mysql
+  labels:
+    app.kubernetes.io/component: volume
+    app.kubernetes.io/name: mysql
 spec:
   accessModes:
     - ReadWriteOnce
@@ -47,23 +58,31 @@ spec:
     requests:
       storage: 5Gi
 {% endif %}
+{% if ACTIVATE_NOTES %}
 ---
 apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:
-  name: openedx-staticfiles
+  name: notes-data
+  labels:
+    app.kubernetes.io/component: volume
+    app.kubernetes.io/name: notes-data
 spec:
   accessModes:
     - ReadWriteOnce
   resources:
     requests:
       storage: 1Gi
+{% endif %}
 {% if ACTIVATE_RABBITMQ %}
 ---
 apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:
   name: rabbitmq
+  labels:
+    app.kubernetes.io/component: volume
+    app.kubernetes.io/name: rabbitmq
 spec:
   accessModes:
     - ReadWriteOnce

tutor/templates/kustomization.yml

@@ -0,0 +1,40 @@
+---
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+- k8s/namespace.yml
+- k8s/deployments.yml
+- k8s/ingress.yml
+- k8s/jobs.yml
+- k8s/services.yml
+- k8s/volumes.yml
+
+# namespace to deploy all Resources to
+namespace: {{ K8S_NAMESPACE }}
+
+# labels added to all Resources
+# https://kubernetes.io/docs/concepts/overview/working-with-objects/common-labels/
+commonLabels:
+  app.kubernetes.io/instance: openedx-{{ ID }}
+  app.kubernetes.io/version: {{ TUTOR_VERSION }}
+  app.kubernetes.io/part-of: openedx
+  app.kubernetes.io/managed-by: tutor
+
+configMapGenerator:
+- name: openedx-settings-lms
+  files:{% for file in "apps/openedx/settings/lms"|walk_templates %}
+  - {{ file }}{% endfor %}
+- name: openedx-settings-cms
+  files:{% for file in "apps/openedx/settings/cms"|walk_templates %}
+  - {{ file }}{% endfor %}
+- name: openedx-config
+  files:{% for file in "apps/openedx/config"|walk_templates %}
+  - {{ file }}{% endfor %}
+- name: nginx-config
+  files:{% for file in "apps/nginx"|walk_templates %}
+  - {{ file }}{% endfor %}
+{% if ACTIVATE_MYSQL %}- name: mysql-config
+  env: apps/mysql/auth.env{% endif %}
+{% if ACTIVATE_NOTES %}- name: notes-settings
+  files:
+    - apps/notes/settings/tutor.py{% endif %}

tutor/templates/scripts/migrate_xqueue.sh

@@ -0,0 +1 @@
+./manage.py migrate

tutor/utils.py

@@ -51,6 +51,15 @@ def reverse_host(domain):
     return ".".join(domain.split(".")[::-1])
 
 
+def walk_files(path):
+    """
+    Iterate on file paths located in directory.
+    """
+    for dirpath, _, filenames in os.walk(path):
+        for filename in filenames:
+            yield os.path.join(dirpath, filename)
+
+
 def docker_run(*command):
     return docker("run", "--rm", "-it", *command)