Make it possible to (de)activate individual data services

Close #114

docs/configuration.rst

@@ -25,21 +25,26 @@ Once the base configuration is created or updated, the environment is automatica
 
 With an up-to-date environment, Tutor is ready to launch an Open edX platform and perform usual operations. Below, we document some of the configuration parameters.
 
+Docker
+------
+
 .. _docker_images:
 
-``DOCKER_IMAGE_*`` Custom Docker images
----------------------------------------
+Custom images
+-------------
+
+- ``DOCKER_IMAGE_OPENEDX`` (default: ``"regis/openedx:hawthorn"``)
+- ``DOCKER_IMAGE_ANDROID`` (default: ``"regis/openedx-android:hawthorn"``)
+- ``DOCKER_IMAGE_FORUM`` (default: ``"regis/openedx-forum:hawthorn"``)
+- ``DOCKER_IMAGE_NOTES`` (default: ``"regis/openedx-notes:hawthorn"``)
+- ``DOCKER_IMAGE_XQUEUE`` (default: ``"regis/openedx-xqueue:hawthorn"``)
 
 These configuration parameters define which image to run for each service.
 
-- ``DOCKER_IMAGE_OPENEDX`` (default: ``regis/openedx:hawthorn``)
-- ``DOCKER_IMAGE_ANDROID`` (default: ``regis/openedx-android:hawthorn``)
-- ``DOCKER_IMAGE_FORUM`` (default: ``regis/openedx-forum:hawthorn``)
-- ``DOCKER_IMAGE_NOTES`` (default: ``regis/openedx-notes:hawthorn``)
-- ``DOCKER_IMAGE_XQUEUE`` (default: ``regis/openedx-xqueue:hawthorn``)
+Custom registry
+---------------
 
-``DOCKER_REGISTRY`` Custom Docker registry
-------------------------------------------
+- ``DOCKER_REGISTRY`` (default: ``""``)
 
 You may want to pull/push images from/to a custom docker registry. For instance, for a registry running on ``localhost:5000``, define::
 
@@ -47,13 +52,58 @@ You may want to pull/push images from/to a custom docker registry. For instance,
 
 (the trailing ``/`` is important)
 
-``ACTIVATE_*`` Optional features
---------------------------------
+Vendor services
+---------------
 
-Some optional features may be activated or deactivated during the interactive configuration step. These features change configuration files (during the ``configure`` step) as well as make targets.
+MySQL
+~~~~~
 
-``ACTIVATE_HTTPS`` SSL/TLS certificates for HTTPS access
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+- ``ACTIVATE_MYSQL`` (default: ``true``)
+- ``MYSQL_HOST`` (default: ``"mysql"``)
+- ``MYSQL_PORT`` (default: ``3306``)
+- ``MYSQL_DATABASE`` (default: ``"openedx"``)
+- ``MYSQL_USERNAME`` (default: ``"openedx"``)
+- ``MYSQL_PASSWORD`` (default: randomly generated): root user password. Note that you are responsible for creating the root user if you are using a managed database.
+
+By default, a running Open edX platform deployed with Tutor includes all necessary 3rd-party services, such as MySQL, MongoDb, etc. But it's also possible to store data on a separate database, such as `Amazon RDS <https://aws.amazon.com/rds/>`_. For instance, to store data on an external MySQL database, set the following configuration::
+
+    ACTIVATE_MYSQL: false
+    MYSQL_HOST: yourhost
+    MYSQL_PASSWORD: <root user password>
+
+Elasticsearch
+~~~~~~~~~~~~~
+
+- ``ELASTICSEARCH_HOST`` (default: ``"elasticsearch"``)
+- ``ELASTICSEARCH_PORT`` (default: ``9200``)
+
+Memcached
+~~~~~~~~~
+
+- ``MEMCACHED_HOST`` (default: ``"memcached"``)
+- ``MEMCACHED_PORT`` (default: ``11211``)
+
+Mongodb
+~~~~~~~
+
+- ``ACTIVATE_MONGODB`` (default: ``true``)
+- ``MONGODB_HOST`` (default: ``"mongodb"``)
+- ``MONGODB_DATABASE`` (default: ``"openedx"``)
+
+Rabbitmq
+~~~~~~~~
+
+- ``RABBITMQ_HOST`` (default: ``"rabbitmq"``)
+
+Optional features
+-----------------
+
+Some optional features may be activated or deactivated during the interactive configuration step (``tutor config save``).
+
+SSL/TLS certificates for HTTPS access
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+- ``ACTIVATE_HTTPS`` (default: ``false``)
 
 By activating this feature, a free SSL/TLS certificate from the `Let's Encrypt <https://letsencrypt.org/>`_ certificate authority will be created for your platform. With this feature, **your platform will no longer be accessible in HTTP**. Calls to http urls will be redirected to https url.
 
@@ -73,8 +123,10 @@ To renew the certificate, run this command once per month::
 
     tutor local https renew
 
-``ACTIVATE_NOTES`` Student notes
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Student notes
+~~~~~~~~~~~~~
+
+- ``ACTIVATE_NOTES`` (default: ``false``)
 
 With `notes <https://edx.readthedocs.io/projects/open-edx-building-and-running-a-course/en/open-release-hawthorn.master/exercises_tools/notes.html?highlight=notes>`_, students can annotate portions of the courseware. 
 
@@ -83,7 +135,9 @@ With `notes <https://edx.readthedocs.io/projects/open-edx-building-and-running-a
 
 You should beware that the ``notes.<LMS_HOST>`` domain name should be activated and point to your server. For instance, if your LMS is hosted at http://myopenedx.com, the notes service should be found at http://notes.myopenedx.com.
 
-``ACTIVATE_XQUEUE`` Xqueue
-~~~~~~~~~~~~~~~~~~~~~~~~~~
+Xqueue
+~~~~~~
+
+- ``ACTIVATE_XQUEUE`` (default: ``false``)
 
 `Xqueue <https://github.com/edx/xqueue>`_ is for grading problems with external services. If you don't know what it is, you probably don't need it.

tutor/images.py

@@ -85,7 +85,19 @@ def openedx_image_names(config, image):
     return [image]
 
 def vendor_image_names(config, image):
-    return VENDOR_IMAGES if image == "all" else [image]
+    if image == "all":
+        images = VENDOR_IMAGES[:]
+        if not config['ACTIVATE_ELASTICSEARCH']:
+            images.remove('elasticsearch')
+        if not config['ACTIVATE_MEMCACHED']:
+            images.remove('memcached')
+        if not config['ACTIVATE_MONGODB']:
+            images.remove('mongodb')
+        if not config['ACTIVATE_MYSQL']:
+            images.remove('mysql')
+        if not config['ACTIVATE_RABBITMQ']:
+            images.remove('rabbitmq')
+    return [image]
 
 images.add_command(build)
 images.add_command(pull)

tutor/k8s.py

@@ -21,7 +21,6 @@ def k8s():
 def quickstart(root):
     click.echo(fmt.title("Interactive platform configuration"))
     tutor_config.save.callback(root, False, [])
-    click.echo(fmt.title("Environment generation"))
     click.echo(fmt.title("Stopping any existing platform"))
     stop.callback()
     click.echo(fmt.title("Starting the platform"))
@@ -32,10 +31,12 @@ def quickstart(root):
 @click.command(help="Run all configured Open edX services")
 @opts.root
 def start(root):
+    config = tutor_config.load(root)
     kubectl_no_fail("create", "-f", tutor_env.pathjoin(root, "k8s", "namespace.yml"))
 
     kubectl("create", "configmap", "nginx-config", "--from-file", tutor_env.pathjoin(root, "apps", "nginx"))
-    kubectl("create", "configmap", "mysql-config", "--from-env-file", tutor_env.pathjoin(root, "apps", "mysql", "auth.env"))
+    if config['ACTIVATE_MYSQL']:
+        kubectl("create", "configmap", "mysql-config", "--from-env-file", tutor_env.pathjoin(root, "apps", "mysql", "auth.env"))
     kubectl("create", "configmap", "openedx-settings-lms", "--from-file", tutor_env.pathjoin(root, "apps", "openedx", "settings", "lms"))
     kubectl("create", "configmap", "openedx-settings-cms", "--from-file", tutor_env.pathjoin(root, "apps", "openedx", "settings", "cms"))
     kubectl("create", "configmap", "openedx-config", "--from-file", tutor_env.pathjoin(root, "apps", "openedx", "config"))

tutor/local.py

@@ -124,6 +124,9 @@ def databases(root):
     ops.migrate(root, run_bash)
 
 def init_mysql(root):
+    config = tutor_config.load(root)
+    if not config['ACTIVATE_MYSQL']:
+        return
     mysql_data_path = tutor_env.data_path(root, "mysql", "mysql")
     if os.path.exists(mysql_data_path):
         return

tutor/templates/config-defaults.yml

@@ -1,4 +1,9 @@
 ---
+ACTIVATE_ELASTICSEARCH: true
+ACTIVATE_MEMCACHED: true
+ACTIVATE_MONGODB: true
+ACTIVATE_MYSQL: true
+ACTIVATE_RABBITMQ: true
 ANDROID_RELEASE_STORE_PASSWORD: "android store password"
 ANDROID_RELEASE_KEY_PASSWORD: "android release key password"
 ANDROID_RELEASE_KEY_ALIAS: "android release key alias"

tutor/templates/k8s/deployments.yml

@@ -116,6 +116,7 @@ spec:
           persistentVolumeClaim:
             claimName: lms-data
 
+{% if ACTIVATE_ELASTICSEARCH %}
 ---
 apiVersion: apps/v1
 kind: Deployment
@@ -150,7 +151,9 @@ spec:
         - name: data
           persistentVolumeClaim:
             claimName: elasticsearch
+{% endif %}
 
+{% if ACTIVATE_MEMCACHED %}
 ---
 apiVersion: apps/v1
 kind: Deployment
@@ -171,7 +174,9 @@ spec:
           image: {{ DOCKER_REGISTRY }}{{ DOCKER_IMAGE_MEMCACHED }}
           ports:
             - containerPort: 11211
+{% endif %}
 
+{% if ACTIVATE_MONGODB %}
 ---
 apiVersion: apps/v1
 kind: Deployment
@@ -199,7 +204,9 @@ spec:
       volumes:
         - name: data
           emptyDir: {}
+{% endif %}
 
+{% if ACTIVATE_MYSQL %}
 ---
 apiVersion: apps/v1
 kind: Deployment
@@ -233,6 +240,7 @@ spec:
         - name: data
           persistentVolumeClaim:
             claimName: mysql
+{% endif %}
 
 ---
 apiVersion: apps/v1
@@ -290,6 +298,7 @@ spec:
           persistentVolumeClaim:
             claimName: lms-data
 
+{% if ACTIVATE_RABBITMQ %}
 ---
 apiVersion: apps/v1
 kind: Deployment
@@ -317,4 +326,4 @@ spec:
         - name: data
           persistentVolumeClaim:
             claimName: rabbitmq
-
+{% endif %}

tutor/templates/k8s/services.yml

@@ -37,6 +37,7 @@ spec:
   selector:
     app: lms
 
+{% if ACTIVATE_ELASTICSEARCH %}
 ---
 apiVersion: v1
 kind: Service
@@ -49,7 +50,9 @@ spec:
       protocol: TCP
   selector:
     app: elasticsearch
+{% endif %}
 
+{% if ACTIVATE_MEMCACHED %}
 ---
 apiVersion: v1
 kind: Service
@@ -62,7 +65,9 @@ spec:
       protocol: TCP
   selector:
     app: memcached
+{% endif %}
 
+{% if ACTIVATE_MONGODB %}
 ---
 apiVersion: v1
 kind: Service
@@ -75,7 +80,9 @@ spec:
       protocol: TCP
   selector:
     app: mongodb
+{% endif %}
 
+{% if ACTIVATE_MYSQL %}
 ---
 apiVersion: v1
 kind: Service
@@ -88,6 +95,7 @@ spec:
       protocol: TCP
   selector:
     app: mysql
+{% endif %}
 
 ---
 apiVersion: v1
@@ -108,6 +116,7 @@ spec:
   selector:
     app: nginx
 
+{% if ACTIVATE_RABBITMQ %}
 ---
 apiVersion: v1
 kind: Service
@@ -120,4 +129,4 @@ spec:
       protocol: TCP
   selector:
     app: rabbitmq
-
+{% endif %}

tutor/templates/k8s/volumes.yml

@@ -22,6 +22,7 @@ spec:
     requests:
       storage: 2Gi
 
+{% if ACTIVATE_ELASTICSEARCH %}
 ---
 apiVersion: v1
 kind: PersistentVolumeClaim
@@ -33,7 +34,9 @@ spec:
   resources:
     requests:
       storage: 2Gi
+{% endif %}
 
+{% if ACTIVATE_MYSQL %}
 ---
 apiVersion: v1
 kind: PersistentVolumeClaim
@@ -45,6 +48,7 @@ spec:
   resources:
     requests:
       storage: 5Gi
+{% endif %}
 
 ---
 apiVersion: v1
@@ -58,6 +62,7 @@ spec:
     requests:
       storage: 1Gi
 
+{% if ACTIVATE_RABBITMQ %}
 ---
 apiVersion: v1
 kind: PersistentVolumeClaim
@@ -69,3 +74,4 @@ spec:
   resources:
     requests:
       storage: 1Gi
+{% endif %}

tutor/templates/local/docker-compose.yml

@@ -3,10 +3,13 @@ services:
 
   ############# External services
 
+  {% if ACTIVATE_MEMCACHED %}
   memcached:
     image: {{ DOCKER_REGISTRY }}{{ DOCKER_IMAGE_MEMCACHED }}
     restart: unless-stopped
+  {% endif %}
 
+  {% if ACTIVATE_MONGODB %}
   mongodb:
     image: {{ DOCKER_REGISTRY }}{{ DOCKER_IMAGE_MONGODB }}
     # Use WiredTiger in all environments, just like at edx.org
@@ -14,7 +17,9 @@ services:
     restart: unless-stopped
     volumes:
       - ../../data/mongodb:/data/db
+  {% endif %}
 
+  {% if ACTIVATE_MYSQL %}
   mysql:
     image: {{ DOCKER_REGISTRY }}{{ DOCKER_IMAGE_MYSQL }}
     command: mysqld --character-set-server=utf8 --collation-server=utf8_general_ci
@@ -22,7 +27,9 @@ services:
     volumes:
       - ../../data/mysql:/var/lib/mysql
     env_file: ../apps/mysql/auth.env
+  {% endif %}
 
+  {% if ACTIVATE_ELASTICSEARCH %}
   elasticsearch:
     image: {{ DOCKER_REGISTRY }}{{ DOCKER_IMAGE_ELASTICSEARCH }}
     environment:
@@ -36,6 +43,7 @@ services:
     restart: unless-stopped
     volumes:
       - ../../data/elasticsearch:/usr/share/elasticsearch/data
+  {% endif %}
 
   openedx-assets:
     image: {{ DOCKER_REGISTRY }}{{ DOCKER_IMAGE_OPENEDX }}
@@ -60,11 +68,13 @@ services:
       - cms
       {% if ACTIVATE_NOTES %}- notes{% endif %}
 
+  {% if ACTIVATE_RABBITMQ %}
   rabbitmq:
     image: {{ DOCKER_REGISTRY }}{{ DOCKER_IMAGE_RABBITMQ }}
     volumes:
       - ../../data/rabbitmq:/var/lib/rabbitmq
     restart: unless-stopped
+  {% endif %}
 
   # Simple SMTP server
   smtp:
@@ -80,8 +90,8 @@ services:
       MONGOHQ_URL: "mongodb://{{ MONGODB_HOST }}/cs_comments_service"
     restart: unless-stopped
     depends_on:
-      - elasticsearch
-      - mongodb
+      {% if ACTIVATE_ELASTICSEARCH %}- elasticsearch{% endif %}
+      {% if ACTIVATE_MONGODB %}- mongodb{% endif %}
 
   ############# LMS and CMS
 
@@ -97,12 +107,12 @@ services:
       - ../apps/openedx/config/:/openedx/config/
       - ../../data/lms:/openedx/data
     depends_on:
-      - elasticsearch
+      {% if ACTIVATE_ELASTICSEARCH %}- elasticsearch{% endif %}
       - forum
-      - memcached
-      - mongodb
-      - mysql
-      - rabbitmq
+      {% if ACTIVATE_MEMCACHED %}- memcached{% endif %}
+      {% if ACTIVATE_MONGODB %}- mongodb{% endif %}
+      {% if ACTIVATE_MYSQL %}- mysql{% endif %}
+      {% if ACTIVATE_RABBITMQ %}- rabbitmq{% endif %}
       - smtp
 
   cms:
@@ -117,11 +127,11 @@ services:
       - ../apps/openedx/config/:/openedx/config/
       - ../../data/cms:/openedx/data
     depends_on:
-      - elasticsearch
-      - memcached
-      - mongodb
-      - mysql
-      - rabbitmq
+      {% if ACTIVATE_ELASTICSEARCH %}- elasticsearch{% endif %}
+      {% if ACTIVATE_MEMCACHED %}- memcached{% endif %}
+      {% if ACTIVATE_MONGODB %}- mongodb{% endif %}
+      {% if ACTIVATE_MYSQL %}- mysql{% endif %}
+      {% if ACTIVATE_RABBITMQ %}- rabbitmq{% endif %}
       - smtp
 
   ############# LMS and CMS workers
@@ -174,7 +184,7 @@ services:
       - ../../data/notes:/openedx/data
     restart: unless-stopped
     depends_on:
-      - mysql
+      {% if ACTIVATE_MYSQL %}- mysql{% endif %}
   {% endif %}
 
   {% if ACTIVATE_XQUEUE %}
@@ -188,7 +198,7 @@ services:
       DJANGO_SETTINGS_MODULE: xqueue.tutor
     restart: unless-stopped
     depends_on:
-      - mysql
+      {% if ACTIVATE_MYSQL %}- mysql{% endif %}
 
   xqueue_consumer:
     image: {{ DOCKER_REGISTRY }}{{ DOCKER_IMAGE_XQUEUE }}
@@ -200,5 +210,5 @@ services:
     restart: unless-stopped
     command: ./manage.py run_consumer
     depends_on:
-      - mysql
+      {% if ACTIVATE_MYSQL %}- mysql{% endif %}
   {% endif %}