diff --git a/CHANGELOG.md b/CHANGELOG.md index fec98e5..9047c15 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,31 @@ Note: Breaking changes between versions are indicated by "💥". ## Unreleased +## v11.0.0 (2020-12-09) + +- 💥[Improvement] Upgrade Open edX to Koa + - 💥 Setting changes: + - The ``ACTIVATE_HTTPS`` setting was renamed to ``ENABLE_HTTPS``. + - Other ``ACTIVATE_*`` variables were all renamed to ``RUN_*``. + - The ``WEB_PROXY`` setting was removed and ``RUN_CADDY`` was added. + - The ``NGINX_HTTPS_PORT`` setting is deprecated. + - Architectural changes: + - Use Caddy as a web proxy for automated SSL/TLS certificate generation: + - Nginx no longer listens to port 443 for https traffic + - The Caddy configuration file comes with a new ``caddyfile`` patch for much simpler SSL/TLS management. + - Configuration files for web proxies are no longer provided. + - Kubernetes deployment no longer requires setting up a custom Ingress resource or custom manager. + - Gunicorn and Whitenoise are replaced by uwsgi: this increases boostrap performance and makes it no longer necessary to mount media folders in the Nginx container. + - Replace memcached and rabbitmq by redis. + - Additional features: + - Make it possible to disable all plugins at once with ``plugins disable all``. + - Add ``tutor k8s wait`` command to wait for a pod to become ready + - Faster, more reliable static assets with local memory caching + - Deprecation: proxy files for Apache and Nginx are no longer provided out of the box. + - Removed plugin `{{ patch (...) }}` statements: + - "https-create", "k8s-ingress-rules", "k8s-ingress-tls-hosts": these are no longer necessary. Instead, declare your app in the "caddyfile" patch. + - "local-docker-compose-nginx-volumes": this patch was primarily used to serve media assets. The recommended is now to serve assets with uwsgi. + ## v10.5.3 (2020-12-09) - [Security] Apply upstream edx-platform [security patch](https://github.com/edx/edx-platform/pull/25782) diff --git a/Makefile b/Makefile index 6f8fc96..355924d 100644 --- a/Makefile +++ b/Makefile @@ -89,10 +89,10 @@ ci-test-bundle: ## Run basic tests on bundle yes "" | ./dist/tutor config save --interactive ./dist/tutor config save ./dist/tutor plugins list - # ./dist/tutor plugins enable discovery ecommerce figures lts minio notes xqueue - ./dist/tutor plugins enable discovery ecommerce lts minio notes xqueue + # ./dist/tutor plugins enable discovery ecommerce figures license minio notes xqueue + ./dist/tutor plugins enable discovery ecommerce license minio notes xqueue ./dist/tutor plugins list - ./dist/tutor lts --help + ./dist/tutor license --help ./releases/github-release: ## Download github-release binary mkdir -p releases/ diff --git a/README.rst b/README.rst index 10b6918..154794b 100644 --- a/README.rst +++ b/README.rst @@ -34,7 +34,7 @@ Tutor: the docker-based Open edX distribution designed for peace of mind **Tutor** is a docker-based `Open edX `_ distribution, both for production and local development. The goal of Tutor is to make it easy to deploy, customize, upgrade and scale Open edX. Tutor is reliable, fast, extensible, and it is already used by dozens of Open edX platforms around the world. -Do you need professional assistance setting up or managing your Open edX platform? Overhang.IO provides online support as part of its `Long Term Support (LTS) offering `__. +Do you need professional assistance setting up or managing your Open edX platform? Overhang.IO provides online support as part of its `Long Term Support (LTS) offering `__. Features -------- @@ -46,7 +46,7 @@ Features * Extensible architecture with `plugins `__ * Works with `Kubernetes `__ * No technical skill required with the `1-click Tutor AWS image `__ -* Professional support and premium plugins available with `Tutor Long Term Support (LTS) `__ +* Amazing plugins available with `Tutor Wizard Edition `__ .. _readme_intro_end: @@ -71,7 +71,7 @@ Extensive documentation is available online: https://docs.tutor.overhang.io/ Support ------- -To get community support, go to the official discussion forums: https://discuss.overhang.io. For official support, please subscribe to a Long Term Support (LTS) license at https://overhang.io/tutor/lts. +To get community support, go to the official discussion forums: https://discuss.overhang.io. For official support, please subscribe to a Long Term Support (LTS) license at https://overhang.io/tutor/pricing. .. _readme_support_end: diff --git a/bin/main.py b/bin/main.py index 7f3ce22..feaf04c 100755 --- a/bin/main.py +++ b/bin/main.py @@ -6,7 +6,7 @@ for plugin_name in [ "discovery", "ecommerce", # "figures", - "lts", + "license", "minio", "notes", "xqueue", diff --git a/docs/configuration.rst b/docs/configuration.rst index 18064b9..d61d949 100644 --- a/docs/configuration.rst +++ b/docs/configuration.rst @@ -40,16 +40,15 @@ With an up-to-date environment, Tutor is ready to launch an Open edX platform an Individual service activation ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -- ``ACTIVATE_LMS`` (default: ``true``) -- ``ACTIVATE_CMS`` (default: ``true``) -- ``ACTIVATE_FORUM`` (default: ``true``) -- ``ACTIVATE_ELASTICSEARCH`` (default: ``true``) -- ``ACTIVATE_MEMCACHED`` (default: ``true``) -- ``ACTIVATE_MONGODB`` (default: ``true``) -- ``ACTIVATE_MYSQL`` (default: ``true``) -- ``ACTIVATE_RABBITMQ`` (default: ``true``) -- ``ACTIVATE_SMTP`` (default: ``true``) -- ``ACTIVATE_HTTPS`` (default: ``false``) +- ``RUN_LMS`` (default: ``true``) +- ``RUN_CMS`` (default: ``true``) +- ``RUN_FORUM`` (default: ``true``) +- ``RUN_ELASTICSEARCH`` (default: ``true``) +- ``RUN_MONGODB`` (default: ``true``) +- ``RUN_MYSQL`` (default: ``true``) +- ``RUN_REDIS`` (default: ``true``) +- ``RUN_SMTP`` (default: ``true``) +- ``ENABLE_HTTPS`` (default: ``false``) Every single Open edX service may be (de)activated at will by these configuration parameters. This is useful if you want, for instance, to distribute the various Open edX services on different servers. @@ -81,32 +80,37 @@ You may want to pull/push images from/to a custom docker registry. For instance, Open edX customisation ~~~~~~~~~~~~~~~~~~~~~~ -- ``OPENEDX_COMMON_VERSION`` (default: ``"open-release/juniper.3"``) +- ``OPENEDX_COMMON_VERSION`` (default: ``"open-release/koa.1"``) This defines the default version that will be pulled from all Open edX git repositories. -- ``OPENEDX_CMS_GUNICORN_WORKERS`` (default: ``2``) -- ``OPENEDX_LMS_GUNICORN_WORKERS`` (default: ``2``) +- ``OPENEDX_CMS_UWSGI_WORKERS`` (default: ``2``) +- ``OPENEDX_LMS_UWSGI_WORKERS`` (default: ``2``) -By default there are 2 `gunicorn worker processes `__ to serve requests for the LMS and the CMS. However, each workers requires upwards of 500 Mb of RAM. You should reduce this value to 1 if your computer/server does not have enough memory. +By default there are 2 `uwsgi worker processes `__ to serve requests for the LMS and the CMS. However, each workers requires upwards of 500 Mb of RAM. You should reduce this value to 1 if your computer/server does not have enough memory. Vendor services ~~~~~~~~~~~~~~~ +Caddy +***** + +- ``RUN_CADDY`` (default: ``true``) + +`Caddy `__ is a web server used in Tutor as a web proxy for the generation of SSL/TLS certificates at runtime. If ``RUN_CADDY`` is set to ``false`` then we assume that SSL termination does not occur in the Caddy container, and thus the ``caddy`` container is not started. + Nginx ***** - ``NGINX_HTTP_PORT`` (default: ``80``) -- ``NGINX_HTTPS_PORT`` (default: ``443``) -- ``WEB_PROXY`` (default: ``false``) -Nginx is used to route web traffic to the various applications and to serve static assets. In case there is another web server in front of the Nginx container (for instance, a web server running on the host or an Ingress controller on Kubernetes), the container exposed ports can be modified. If ``WEB_PROXY`` is set to ``true`` then we assume that SSL termination does not occur in the Nginx container. +Nginx is used to route web traffic to the various applications and to serve static assets. When ``RUN_CADDY`` is false, the ``NGINX_HTTP_PORT`` is exposed on the host. MySQL ***** -- ``ACTIVATE_MYSQL`` (default: ``true``) +- ``RUN_MYSQL`` (default: ``true``) - ``MYSQL_HOST`` (default: ``"mysql"``) - ``MYSQL_PORT`` (default: ``3306``) - ``MYSQL_ROOT_USERNAME`` (default: ``"root"``) @@ -114,7 +118,7 @@ MySQL By default, a running Open edX platform deployed with Tutor includes all necessary 3rd-party services, such as MySQL, MongoDb, etc. But it's also possible to store data on a separate database, such as `Amazon RDS `_. For instance, to store data on an external MySQL database, set the following configuration:: - ACTIVATE_MYSQL: false + RUN_MYSQL: false MYSQL_HOST: yourhost MYSQL_ROOT_USERNAME: MYSQL_ROOT_PASSWORD: @@ -127,34 +131,31 @@ Elasticsearch - ``ELASTICSEARCH_PORT`` (default: ``9200``) - ``ELASTICSEARCH_HEAP_SIZE`` (default: ``"1g"``) -Memcached -********* - -- ``MEMCACHED_HOST`` (default: ``"memcached"``) -- ``MEMCACHED_PORT`` (default: ``11211``) - Mongodb ******* -- ``ACTIVATE_MONGODB`` (default: ``true``) +- ``RUN_MONGODB`` (default: ``true``) - ``MONGODB_HOST`` (default: ``"mongodb"``) - ``MONGODB_DATABASE`` (default: ``"openedx"``) - ``MONGODB_PORT`` (default: ``27017``) - ``MONGODB_USERNAME`` (default: ``""``) - ``MONGODB_PASSWORD`` (default: ``""``) -Rabbitmq -******** +Redis +***** -- ``ACTIVATE_RABBITMQ`` (default: ``true``) -- ``RABBITMQ_HOST`` (default: ``"rabbitmq"``) -- ``RABBITMQ_USERNAME`` (default: ``""``) -- ``RABBITMQ_PASSWORD`` (default: ``""``) +- ``RUN_REDIS`` (default: ``true``) +- ``REDIS_HOST`` (default: ``"redis"``) +- ``REDIS_HOST`` (default: ``6379``) +- ``REDIS_USERNAME`` (default: ``""``) +- ``REDIS_PASSWORD`` (default: ``""``) + +Note that Redis has replaced Rabbitmq as the Celery message broker since Tutor v11.0.0. SMTP **** -- ``ACTIVATE_SMTP`` (default: ``true``) +- ``RUN_SMTP`` (default: ``true``) - ``SMTP_HOST`` (default: ``"smtp"``) - ``SMTP_PORT`` (default: ``25``) - ``SMTP_USERNAME`` (default: ``""``) @@ -167,7 +168,7 @@ Note that the SMTP server shipped with Tutor by default does not implement TLS. SSL/TLS certificates for HTTPS access ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -- ``ACTIVATE_HTTPS`` (default: ``false``) +- ``ENABLE_HTTPS`` (default: ``false``) By activating this feature, a free SSL/TLS certificate from the `Let's Encrypt `_ certificate authority will be created for your platform. With this feature, **your platform will no longer be accessible in HTTP**. Calls to http urls will be redirected to https url. @@ -179,15 +180,7 @@ The following DNS records must exist and point to your server:: Thus, **this feature will (probably) not work in development** because the DNS records will (probably) not point to your development machine. -To create the certificate manually, run:: - - tutor local https create - -To renew the certificate, run this command once per month:: - - tutor local stop nginx - tutor local https renew - tutor local start -d +The SSL/TLS certificates will automatically be generated and updated by the Caddy proxy server container at runtime. Thus, as of v11.0.0 you no longer have to generate the certificates manually. .. _customise: @@ -214,7 +207,7 @@ openedx Docker Image build arguments When building the "openedx" Docker image, it is possible to specify a few `arguments `__: - ``EDX_PLATFORM_REPOSITORY`` (default: ``"https://github.com/edx/edx-platform.git"``) -- ``EDX_PLATFORM_VERSION`` (default: ``"open-release/juniper.3"``) +- ``EDX_PLATFORM_VERSION`` (default: ``"open-release/koa.1"``) - ``EDX_PLATFORM_VERSION_DATE`` (default: ``"20200227"``) - ``NPM_REGISTRY`` (default: ``"https://registry.npmjs.org/"``) @@ -286,16 +279,16 @@ Note that your edx-platform version must be a fork of the latest release **tag** If you don't create your fork from this tag, you *will* have important compatibility issues with other services. In particular: -- Do not try to run a fork from an older (pre-Juniper) version of edx-platform: this will simply not work. +- Do not try to run a fork from an older (pre-Koa) version of edx-platform: this will simply not work. - Do not try to run a fork from the edx-platform master branch: there is a 99% probability that it will fail. -- Do not try to run a fork from the open-release/juniper.master branch: Tutor will attempt to apply security and bug fix patches that might already be included in the open-release/juniper.master but which were not yet applied to the latest release tag. Patch application will thus fail if you base your fork from the open-release/juniper.master branch. +- Do not try to run a fork from the open-release/koa.master branch: Tutor will attempt to apply security and bug fix patches that might already be included in the open-release/koa.master but which were not yet applied to the latest release tag. Patch application will thus fail if you base your fork from the open-release/koa.master branch. .. _i18n: Adding custom translations ~~~~~~~~~~~~~~~~~~~~~~~~~~ -If you are not running Open edX in English, chances are that some strings will not be properly translated. In most cases, this is because not enough contributors have helped translate Open edX in your language. It happens! With Tutor, available translated languages include those that come bundled with `edx-platform `__ as well as those from `openedx-i18n `__. +If you are not running Open edX in English, chances are that some strings will not be properly translated. In most cases, this is because not enough contributors have helped translate Open edX in your language. It happens! With Tutor, available translated languages include those that come bundled with `edx-platform `__ as well as those from `openedx-i18n `__. Tutor offers a relatively simple mechanism to add custom translations to the openedx Docker image. You should create a folder that corresponds to your language code in the "build/openedx/locale" folder of the Tutor environment. This folder should contain a "LC_MESSAGES" folder. For instance:: @@ -308,9 +301,9 @@ Then, add a "django.po" file there that will contain your custom translations:: msgid "String to translate" msgstr "你翻译的东西 la traduction de votre bidule" -The "String to translate" part should match *exactly* the string that you would like to translate. You cannot make it up! The best way to find this string is to copy-paste it from the `upstream django.po file for the English language `__. +The "String to translate" part should match *exactly* the string that you would like to translate. You cannot make it up! The best way to find this string is to copy-paste it from the `upstream django.po file for the English language `__. -If you cannot find the string to translate in this file, then it means that you are trying to translate a string that is used in some piece of javascript code. Those strings are stored in a different file named "djangojs.po". You can check it out `in the edx-platform repo as well `__. Your custom javascript strings should also be stored in a "djangojs.po" file that should be placed in the same directory. +If you cannot find the string to translate in this file, then it means that you are trying to translate a string that is used in some piece of javascript code. Those strings are stored in a different file named "djangojs.po". You can check it out `in the edx-platform repo as well `__. Your custom javascript strings should also be stored in a "djangojs.po" file that should be placed in the same directory. To recap, here is an example. To translate a few strings in French, both from django.po and djangojs.po, we would have the following file hierarchy:: diff --git a/docs/dev.rst b/docs/dev.rst index c24ba0f..01b1fd4 100644 --- a/docs/dev.rst +++ b/docs/dev.rst @@ -25,7 +25,7 @@ This ``openedx-dev`` development image differs from the ``openedx`` production i - The user that runs inside the container has the same UID as the user on the host, in order to avoid permission problems inside mounted volumes (and in particular in the edx-platform repository). - Additional python and system requirements are installed for convenient debugging: `ipython `__, `ipdb `__, vim, telnet. -- The edx-platform `development requirements `__ are installed. +- The edx-platform `development requirements `__ are installed. Since the ``openedx-dev`` is based upon the ``openedx`` docker image, it should be re-built every time the ``openedx`` docker image is modified. @@ -68,11 +68,11 @@ If you don't want to rewrite this option every time, you can define a command al alias tutor-dev-run-lms="tutor dev run -v /path/to/edx-platform:/openedx/edx-platform lms" For technical reasons, the ``-v`` option is only supported for the ``run`` and ``runserver`` commands. With these commands, only one service is started. But there are cases where you may want to launch and debug a complete Open edX platform with ``tutor dev start`` and mount a custom edx-platform fork. For instance, this might be needed when testing the interaction between multiple services. To do so, you should create a ``docker-compose.override.yml`` file that will specify a custom volume to be used with all ``dev`` commands:: - + vim "$(tutor config printroot)/env/dev/docker-compose.override.yml" Then, add the following content:: - + version: "3.7" services: lms: @@ -90,7 +90,7 @@ Then, add the following content:: This override file will be loaded when running any ``tutor dev ..`` command. The edx-platform repo mounted at the specified path will be automaticall mounted inside all LMS and CMS containers. With this file, you should no longer specify the ``-v`` option from the command line with the ``run`` or ``runserver`` commands. -**Note:** containers are built on the Juniper release. If you are working on a different version of Open edX, you will have to rebuild the ``openedx`` docker images with the version. See the :ref:`fork edx-platform section `. +**Note:** containers are built on the Koa release. If you are working on a different version of Open edX, you will have to rebuild the ``openedx`` docker images with the version. See the :ref:`fork edx-platform section `. Prepare the edx-platform repo ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/docs/install.rst b/docs/install.rst index 634e7fd..17a90de 100644 --- a/docs/install.rst +++ b/docs/install.rst @@ -63,7 +63,7 @@ Installing from source pip install -e . .. _cloud_install: - + Zero-click AWS installation ~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -78,9 +78,9 @@ Upgrading With Tutor, it is very easy to upgrade to a more recent Open edX or Tutor release. Just install the latest ``tutor`` version (using either methods above) and run the ``quickstart`` command again. If you have :ref:`customised ` your docker images, you will have to re-build them prior to running ``quickstart``. -``quickstart`` should take care of automatically running the upgrade process. If for some reason you need to *manually* upgrade from an Open edX release to the next, you should run ``tutor local upgrade``. For instance, to upgrade from Ironwood to Juniper, run:: +``quickstart`` should take care of automatically running the upgrade process. If for some reason you need to *manually* upgrade from an Open edX release to the next, you should run ``tutor local upgrade``. For instance, to upgrade from Juniper to Koa, run:: - tutor local upgrade --from=ironwood + tutor local upgrade --from=juniper .. _autocomplete: diff --git a/docs/k8s.rst b/docs/k8s.rst index 3889da3..087d93d 100644 --- a/docs/k8s.rst +++ b/docs/k8s.rst @@ -27,57 +27,10 @@ The Kubernetes cluster should have at least 4Gb of RAM on each node. When runnin .. image:: img/virtualbox-minikube-system.png :alt: Virtualbox memory settings for Minikube -Ingress controller -~~~~~~~~~~~~~~~~~~ +Ingress controller and SSL/TLS certificates +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -In order to access your platform, you will have to setup an Ingress controller. Instructions vary for each cloud provider. To deploy an Nginx Ingress controller, it might be as simple as running:: - - kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.30.0/deploy/static/mandatory.yaml - kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.30.0/deploy/static/provider/cloud-generic.yaml - -See the `official instructions `_ for more details. - - -.. warning:: - By default, Tutor does *not* launch an Ingress resource or TLS/SSL certificate issuer for you. There are many different ways to create an Ingress resource and issue certificates in a Kubernetes cluster, and it's not the responsibility of Tutor to make this decision. However, Tutor comes with a ready-to-run configuration for an Nginx-based Ingress ressource and a `cert-manager `__ Issuer that delivers `Let's Encrypt `__ certificates. You may examine the configuration in ``$(tutor config printroot)/env/k8s/ingress.yml``. If you are happy with this configuration, you may apply it with:: - - kubectl apply -k $(tutor config printroot)/env --selector="app.kubernetes.io/component in (ingress, issuer)" - -On Minikube, run:: - - minikube addons enable ingress - -With Kubernetes, your Open edX platform will *not* be available at localhost. Instead, you will have to access your platform with the domain names you specified for the LMS and the CMS. To do so on a local computer, you will need to add the following line to /etc/hosts:: - - MINIKUBEIP yourdomain.com studio.yourdomain.com preview.yourdomain.com notes.yourdomain.com - -where ``MINIKUBEIP`` should be replaced by the result of the command ``minikube ip``. - -cert-manager for TLS certificates -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Tutor relies on `cert-manager `_ to generate TLS certificates for HTTPS access. In order to activate HTTPS support, you will have to install cert-manager yourself. To do so, follow the `instructions from the official documentation `_. It might be as simple as running:: - - kubectl apply --validate=false -f https://github.com/jetstack/cert-manager/releases/download/v0.14.0/cert-manager.yaml - -If you decide to enable HTTPS certificates, you will also have to set ``WEB_PROXY=true`` in the platform configuration, because the SSL/TLS termination will not occur in the Nginx container, but in the Ingress controller. To do so, run:: - - tutor config save --set WEB_PROXY=true - -Note that this configuration might conflict with a local installation. - -.. warning:: - On DigitalOcean, there is currently a bug that prevents certificate issuers from successfully fetching TLS certificates from Let's Encrypt. A workaround consists in adding a custom annotation to the "ingress-nginx" service:: - - kubectl -n ingress-nginx patch service ingress-nginx -p \ - '{"metadata": {"annotations": {"service.beta.kubernetes.io/do-loadbalancer-hostname": "YOURLMSHOSTHERE"}}}' - - Sources: - - * https://www.digitalocean.com/community/questions/how-do-i-correct-a-connection-timed-out-error-during-http-01-challenge-propagation-with-cert-manager - * https://www.digitalocean.com/community/questions/pod-unable-to-curl-loadbalancer - * https://github.com/jetstack/cert-manager/issues/863#issuecomment-567062996 - * https://github.com/digitalocean/digitalocean-cloud-controller-manager/blob/master/docs/controllers/services/examples/README.md#accessing-pods-over-a-managed-load-balancer-from-inside-the-cluster +As of Tutor v11, it is no longer required to setup an Ingress controller to access your platform. Instead Caddy exposes a LoadBalancer service and SSL/TLS certificates are transparently generated at runtime. S3-like object storage with `MinIO `_ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/docs/local.rst b/docs/local.rst index fa589b9..a3ec0d1 100644 --- a/docs/local.rst +++ b/docs/local.rst @@ -148,9 +148,9 @@ To update the course search index, run:: Reloading Open edX settings ~~~~~~~~~~~~~~~~~~~~~~~~~~~ -After modifying Open edX settings, for instance when running ``tutor config save``, you will want to restart the web processes of the LMS and the CMS to take into account those new settings. It is possible to simply restart the whole platform (with ``tutor local reboot``) or just a single service (``tutor local restart lms``) but that is overkill. A quicker alternative is to send the HUP signal to the gunicorn processes running inside the containers. The "openedx" Docker image comes with a convenient script that does just that. To run it, execute:: +After modifying Open edX settings, for instance when running ``tutor config save``, you will want to restart the web processes of the LMS and the CMS to take into account those new settings. It is possible to simply restart the whole platform (with ``tutor local reboot``) or just a single service (``tutor local restart lms``) but that is overkill. A quicker alternative is to send the HUP signal to the uwsgi processes running inside the containers. The "openedx" Docker image comes with a convenient script that does just that. To run it, execute:: - tutor local exec lms reload-gunicorn + tutor local exec lms reload-uwsgi .. _portainer: @@ -182,26 +182,9 @@ Running Open edX behind a web proxy The containerized web server (nginx) needs to listen to ports 80 and 443 on the host. If there is already a webserver running on the host, such as Apache or Nginx, the nginx container will not be able to start. Tutor supports running behind a web proxy. To do so, add the following configuration:: - tutor config save --set WEB_PROXY=true --set NGINX_HTTP_PORT=81 --set NGINX_HTTPS_PORT=444 + tutor config save --set RUN_CADDY=false --set NGINX_HTTP_PORT=81 -In this example, the nginx container ports would be mapped to 81 and 444, instead of 80 and 443. You must then configure the web proxy on the host. Basic configuration files are provided by Tutor which can be used directly by your web proxy. - -For nginx:: - - sudo ln -s "$(tutor config printroot)/env/local/proxy/nginx/openedx.conf" /etc/nginx/sites-enabled/ - sudo systemctl reload nginx - -For apache:: - - sudo a2enmod proxy - sudo a2enmod proxy_http - sudo ln -s "$(tutor config printroot)/env/local/proxy/apache2/openedx.conf" /etc/apache2/sites-enabled/ - sudo systemctl reload apache2 - -If you have configured your platform to use SSL/TLS certificates for HTTPS access, the generation and renewal of certificates will not be managed by Tutor: you are supposed to take care of it yourself. Suggestions for generating and renewing these certificates with `Let's Encrypt `_ are given by:: - - tutor local https create - tutor local https renew +In this example, the nginx container port would be mapped to 81 instead of 80. You must then configure the web proxy on the host. As of v11.0.0, configuration files are no longer provided for automatic configuration of your web proxy. Basically, you should setup a reverse proxy to `localhost:NGINX_HTTP_PORT` from the following hosts: LMS_HOST, preview.LMS_HOST, CMS_HOST, as well as any additional host exposed by your plugins. Running multiple Open edX platforms on a single server ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -210,7 +193,7 @@ With Tutor, it is easy to run multiple Open edX instances on a single server. To - ``TUTOR_ROOT``: so that configuration, environment and data are not mixed up between platforms. - ``LOCAL_PROJECT_NAME``: the various docker-compose projects cannot share the same name. -- ``NGINX_HTTP_PORT``, ``NGINX_HTTPS_PORT``: ports cannot be shared by two different containers. +- ``NGINX_HTTP_PORT``: ports cannot be shared by two different containers. - ``LMS_HOST``, ``CMS_HOST``: the different platforms must be accessible from different domain (or subdomain) names. In addition, a web proxy must be setup on the host, as described :ref:`above `. @@ -219,13 +202,13 @@ As an example, here is how to launch two different platforms, with nginx running # platform 1 export TUTOR_ROOT=~/openedx/site1 - tutor config save --interactive --set WEB_PROXY=true --set LOCAL_PROJECT_NAME=tutor_site1 --set NGINX_HTTP_PORT=81 --set NGINX_HTTPS_PORT=481 + tutor config save --interactive --set RUN_CADDY=false --set LOCAL_PROJECT_NAME=tutor_site1 --set NGINX_HTTP_PORT=81 tutor local quickstart sudo ln -s "$(tutor config printroot)/env/local/proxy/nginx/openedx.conf" /etc/nginx/sites-enabled/site1.conf # platform 2 export TUTOR_ROOT=~/openedx/site2 - tutor config save --interactive --set WEB_PROXY=true --set LOCAL_PROJECT_NAME=tutor_site2 --set NGINX_HTTP_PORT=82 --set NGINX_HTTPS_PORT=482 + tutor config save --interactive --set RUN_CADDY=false --set LOCAL_PROJECT_NAME=tutor_site2 --set NGINX_HTTP_PORT=82 tutor local quickstart sudo ln -s "$(tutor config printroot)/env/local/proxy/nginx/openedx.conf" /etc/nginx/sites-enabled/site2.conf diff --git a/docs/quickstart.rst b/docs/quickstart.rst index 9e56633..7d20341 100644 --- a/docs/quickstart.rst +++ b/docs/quickstart.rst @@ -18,7 +18,7 @@ Yes :) This is what happens when you run ``tutor local quickstart``: 2. Configuration files are generated from templates. 3. Docker images are downloaded. 4. Docker containers are provisioned. -5. A full, production-ready Open edX platform (`Juniper `__ release) is run with docker-compose. +5. A full, production-ready Open edX platform (`Koa `__ release) is run with docker-compose. The whole procedure should require less than 10 minutes, on a server with a good bandwidth. Note that your host environment will not be affected in any way, since everything runs inside docker containers. Root access is not even necessary. diff --git a/docs/troubleshooting.rst b/docs/troubleshooting.rst index de1c581..7cbe8d9 100644 --- a/docs/troubleshooting.rst +++ b/docs/troubleshooting.rst @@ -17,7 +17,7 @@ What should you do if you have a problem? 6. If you have a technical background, you may try to decide if the issue is related to Open edX or if it's specific to Tutor. In the latter case, you are most welcome to open an `issue on Github `_. **Please follow the instructions from the issue template!!!** Your issue will be examined in all cases, but you can make our life much easier by giving us as much background information as possible. -Do you need professional assistance with your tutor-managed Open edX platform? Overhang.IO offers online support as part of its `Long Term Support (LTS) offering `__. +Do you need professional assistance with your tutor-managed Open edX platform? Overhang.IO offers online support as part of its `Long Term Support (LTS) offering `__. .. _logging: diff --git a/requirements/plugins.txt b/requirements/plugins.txt index 1b72b83..cb282d8 100644 --- a/requirements/plugins.txt +++ b/requirements/plugins.txt @@ -1,7 +1,7 @@ tutor-discovery tutor-ecommerce #tutor-figures -tutor-lts +tutor-license tutor-minio tutor-notes tutor-xqueue \ No newline at end of file diff --git a/tests/test_config.py b/tests/test_config.py index e73a671..bdd3f41 100644 --- a/tests/test_config.py +++ b/tests/test_config.py @@ -74,7 +74,7 @@ class ConfigTests(unittest.TestCase): self.assertEqual("studio.{{ LMS_HOST }}", defaults["CMS_HOST"]) def test_is_service_activated(self): - config = {"ACTIVATE_SERVICE1": True, "ACTIVATE_SERVICE2": False} + config = {"RUN_SERVICE1": True, "RUN_SERVICE2": False} self.assertTrue(tutor_config.is_service_activated(config, "service1")) self.assertFalse(tutor_config.is_service_activated(config, "service2")) diff --git a/tests/test_env.py b/tests/test_env.py index a544b8b..baa424d 100644 --- a/tests/test_env.py +++ b/tests/test_env.py @@ -85,11 +85,11 @@ class EnvTests(unittest.TestCase): with tempfile.TemporaryDirectory() as root: config = tutor_config.load_current(root, defaults) tutor_config.merge(config, defaults) - config["ACTIVATE_HTTPS"] = True + config["ENABLE_HTTPS"] = True with unittest.mock.patch.object(fmt, "STDOUT"): env.save(root, config) - with open(os.path.join(root, "env", "apps", "nginx", "lms.conf")) as f: - self.assertIn("ssl", f.read()) + with open(os.path.join(root, "env", "apps", "caddy", "Caddyfile")) as f: + self.assertIn("www.myopenedx.com {", f.read()) def test_patch(self): patches = {"plugin1": "abcd", "plugin2": "efgh"} diff --git a/tutor.spec b/tutor.spec index d02198c..f089dab 100644 --- a/tutor.spec +++ b/tutor.spec @@ -23,7 +23,7 @@ for entrypoint in pkg_resources.iter_entry_points("tutor.plugin.v0"): path = os.path.join(plugin_root, folder) if os.path.exists(path): datas.append((path, os.path.join(plugin_root_module_name, folder))) -# Fix lts import: if we don't declare some modules, pyinstaller does not find them +# Fix license import: if we don't declare some modules, pyinstaller does not find them hidden_imports.append("tutorlts.__about__") hidden_imports.append("Crypto.Cipher.AES") hidden_imports.append("Crypto.Cipher.PKCS1_OAEP") diff --git a/tutor/__about__.py b/tutor/__about__.py index 8cd7f41..5b46116 100644 --- a/tutor/__about__.py +++ b/tutor/__about__.py @@ -1 +1 @@ -__version__ = "10.5.3" +__version__ = "11.0.0" diff --git a/tutor/commands/android.py b/tutor/commands/android.py index fd5c0a7..a268665 100644 --- a/tutor/commands/android.py +++ b/tutor/commands/android.py @@ -1,5 +1,7 @@ import click +from .compose import ScriptRunner +from .local import LocalContext from .. import config as tutor_config from .. import env as tutor_env from .. import fmt @@ -42,12 +44,8 @@ cp OpenEdXMobile/build/outputs/apk/prod/{apk_folder}/*.apk /openedx/data/""" def docker_run(root, command): config = tutor_config.load(root) - utils.docker_run( - "--volume={}:/openedx/config/".format(tutor_env.pathjoin(root, "android")), - "--volume={}:/openedx/data/".format(tutor_env.data_path(root, "android")), - config["DOCKER_IMAGE_ANDROID"], - command, - ) + runner = ScriptRunner(root, config, LocalContext.docker_compose) + runner.run_job("android", command) android.add_command(build) diff --git a/tutor/commands/cli.py b/tutor/commands/cli.py index f64ef70..d6a3bd3 100755 --- a/tutor/commands/cli.py +++ b/tutor/commands/cli.py @@ -36,6 +36,8 @@ def main(): cli.add_command(plugins_command) add_plugin_commands(cli) cli() # pylint: disable=no-value-for-parameter + except KeyboardInterrupt: + pass except exceptions.TutorError as e: fmt.echo_error("Error: {}".format(e.args[0])) sys.exit(1) diff --git a/tutor/commands/compose.py b/tutor/commands/compose.py index 4a49fa2..0e346f2 100644 --- a/tutor/commands/compose.py +++ b/tutor/commands/compose.py @@ -120,9 +120,9 @@ def restart(context, services): else: for service in services: if service == "openedx": - if config["ACTIVATE_LMS"]: + if config["RUN_LMS"]: command += ["lms", "lms-worker"] - if config["ACTIVATE_CMS"]: + if config["RUN_CMS"]: command += ["cms", "cms-worker"] else: command.append(service) diff --git a/tutor/commands/images.py b/tutor/commands/images.py index 18d99cd..63bbaf7 100644 --- a/tutor/commands/images.py +++ b/tutor/commands/images.py @@ -8,6 +8,15 @@ from .. import utils BASE_IMAGE_NAMES = ["openedx", "forum", "android"] DEV_IMAGE_NAMES = ["openedx-dev"] +VENDOR_IMAGES = [ + "caddy", + "elasticsearch", + "mongodb", + "mysql", + "nginx", + "redis", + "smtp", +] @click.group(name="images", short_help="Manage docker images") @@ -131,17 +140,9 @@ def all_image_names(config): def vendor_image_names(config): - vendor_images = [ - "elasticsearch", - "memcached", - "mongodb", - "mysql", - "nginx", - "rabbitmq", - "smtp", - ] - for image in vendor_images[:]: - if not config.get("ACTIVATE_" + image.upper(), True): + vendor_images = VENDOR_IMAGES[:] + for image in VENDOR_IMAGES: + if not config.get("RUN_" + image.upper(), True): vendor_images.remove(image) return vendor_images diff --git a/tutor/commands/k8s.py b/tutor/commands/k8s.py index 9d39030..6bca266 100644 --- a/tutor/commands/k8s.py +++ b/tutor/commands/k8s.py @@ -24,12 +24,15 @@ def k8s(): def quickstart(context, non_interactive): click.echo(fmt.title("Interactive platform configuration")) config = interactive_config.update(context.root, interactive=(not non_interactive)) - if config["ACTIVATE_HTTPS"] and not config["WEB_PROXY"]: + if not config["RUN_CADDY"]: fmt.echo_alert( - "Potentially invalid configuration: ACTIVATE_HTTPS=true WEB_PROXY=false\n" - "You should either disable HTTPS support or configure your platform to use" - " a web proxy. See the Kubernetes section in the Tutor documentation for" - " more information." + "Potentially invalid configuration: RUN_CADDY=false\n" + "This setting might have been defined because you previously set WEB_PROXY=true. This is no longer" + " necessary in order to get Tutor to work on Kubernetes. In Tutor v11+ a Caddy-based load balancer is" + " provided out of the box to handle SSL/TLS certificate generation at runtime. If you disable this" + " service, you will have to configure an Ingress resource and a certificate manager yourself to redirect" + " traffic to the nginx service. See the Kubernetes section in the Tutor documentation for more" + " information." ) click.echo(fmt.title("Updating the current environment")) tutor_env.save(context.root, config) @@ -37,6 +40,17 @@ def quickstart(context, non_interactive): start.callback() click.echo(fmt.title("Database creation and migrations")) init.callback(limit=None) + fmt.echo_info( + """Your Open edX platform is ready and can be accessed at the following urls: + + {http}://{lms_host} + {http}://{cms_host} + """.format( + http="https" if config["ENABLE_HTTPS"] else "http", + lms_host=config["LMS_HOST"], + cms_host=config["CMS_HOST"], + ) + ) @click.command(help="Run all configured Open edX services") @@ -60,13 +74,14 @@ def start(context): "--selector", "app.kubernetes.io/component=volume", ) - # Create everything else except jobs, ingress and issuer + # Create everything else except jobs utils.kubectl( "apply", "--kustomize", tutor_env.pathjoin(context.root), "--selector", - "app.kubernetes.io/component notin (job, ingress, issuer)", + # Here use `notin (job, xxx)` when there are other components to ignore + "app.kubernetes.io/component!=job", ) @@ -77,7 +92,7 @@ def stop(context): utils.kubectl( "delete", *resource_selector(config), - "deployments,services,ingress,configmaps,jobs", + "deployments,services,configmaps,jobs", ) @@ -201,6 +216,14 @@ def logs(context, container, follow, tail, service): utils.kubectl(*command) +@click.command(help="Wait for a pod to become ready") +@click.argument("name") +@click.pass_obj +def wait(context, name): + config = tutor_config.load(context.root) + wait_for_pod_ready(config, name) + + @click.command(help="Upgrade from a previous Open edX named release") @click.option( "--from", "from_version", default="ironwood", type=click.Choice(["ironwood"]) @@ -209,15 +232,26 @@ def logs(context, container, follow, tail, service): def upgrade(context, from_version): config = tutor_config.load(context.root) - if from_version == "ironwood": - if not config["ACTIVATE_MONGODB"]: - fmt.echo_info( - "You are not running MongDB (ACTIVATE_MONGODB=false). It is your " - "responsibility to upgrade your MongoDb instance to v3.6. There is " - "nothing left to do." - ) - return - message = """Automatic release upgrade is unsupported in Kubernetes. To upgrade from Ironwood, you should upgrade your MongoDb cluster from v3.2 to v3.6. You should run something similar to: + running_version = from_version + if running_version == "ironwood": + upgrade_from_ironwood(config) + running_version = "juniper" + + if running_version == "juniper": + + running_version = "koa" + + +def upgrade_from_ironwood(config): + if not config["RUN_MONGODB"]: + fmt.echo_info( + "You are not running MongDB (RUN_MONGODB=false). It is your " + "responsibility to upgrade your MongoDb instance to v3.6. There is " + "nothing left to do to upgrade from Ironwood." + ) + return + message = """Automatic release upgrade is unsupported in Kubernetes. To upgrade from Ironwood, you should upgrade +your MongoDb cluster from v3.2 to v3.6. You should run something similar to: # Upgrade from v3.2 to v3.4 tutor k8s stop @@ -232,7 +266,27 @@ def upgrade(context, from_version): tutor k8s exec mongodb mongo --eval 'db.adminCommand({ setFeatureCompatibilityVersion: "3.6" })' tutor config save --unset DOCKER_IMAGE_MONGODB""" - fmt.echo_info(message) + fmt.echo_info(message) + + +def upgrade_from_juniper(config): + if not config["RUN_MYSQL"]: + fmt.echo_info( + "You are not running MySQL (RUN_MYSQL=false). It is your " + "responsibility to upgrade your MySQL instance to v5.7. There is " + "nothing left to do to upgrade from Juniper." + ) + return + + message = """Automatic release upgrade is unsupported in Kubernetes. To upgrade from Juniper, you should upgrade +your MySQL database from v5.6 to v5.7. You should run something similar to: + + tutor k8s start + tutor k8s exec mysql bash -e -c "mysql_upgrade \ + -u $(tutor config printvalue MYSQL_ROOT_USERNAME) \ + --password='$(tutor config printvalue MYSQL_ROOT_PASSWORD)' +""" + fmt.echo_info(message) class K8sClients: @@ -329,12 +383,16 @@ class K8sScriptRunner(scripts.BaseRunner): job["metadata"]["name"] = job_name job["metadata"].setdefault("labels", {}) job["metadata"]["labels"]["app.kubernetes.io/name"] = job_name - job["spec"]["template"]["spec"]["containers"][0]["args"] = [ - "sh", - "-e", - "-c", - command, - ] + # Define k8s entrypoint/args + shell_command = ["sh", "-e", "-c"] + if job["spec"]["template"]["spec"]["containers"][0].get("command") == []: + # Empty "command" (aka: entrypoint) might not be taken into account by jobs, so we need to manually + # override the entrypoint. We do not do this for every job, because some entrypoints are actually useful. + job["spec"]["template"]["spec"]["containers"][0]["command"] = shell_command + container_args = [command] + else: + container_args = shell_command + [command] + job["spec"]["template"]["spec"]["containers"][0]["args"] = container_args job["spec"]["backoffLimit"] = 1 job["spec"]["ttlSecondsAfterFinished"] = 3600 # Save patched job to "jobs.yml" file @@ -429,4 +487,5 @@ k8s.add_command(importdemocourse) k8s.add_command(settheme) k8s.add_command(exec_command) k8s.add_command(logs) +k8s.add_command(wait) k8s.add_command(upgrade) diff --git a/tutor/commands/local.py b/tutor/commands/local.py index 349d3fe..27b6b85 100644 --- a/tutor/commands/local.py +++ b/tutor/commands/local.py @@ -1,5 +1,4 @@ import os -from textwrap import indent import click @@ -57,8 +56,6 @@ def quickstart(context, non_interactive, pullimages_): ) click.echo(fmt.title("Stopping any existing platform")) compose.stop.callback([]) - click.echo(fmt.title("HTTPS certificates generation")) - https_create.callback() if pullimages_: click.echo(fmt.title("Docker image updates")) compose.dc_command.callback(["pull"]) @@ -75,95 +72,19 @@ Your Open edX platform is ready and can be accessed at the following urls: {http}://{lms_host} {http}://{cms_host} """.format( - http="https" if config["ACTIVATE_HTTPS"] else "http", + http="https" if config["ENABLE_HTTPS"] else "http", lms_host=config["LMS_HOST"], cms_host=config["CMS_HOST"], ) ) -@click.group(help="Manage https certificates") -def https(): - pass - - -@click.command(help="Create https certificates", name="create") -@click.pass_obj -def https_create(context): - """ - Note: there are a couple issues with https certificate generation. - 1. Certificates are generated and renewed by using port 80, which is not necessarily open. - a. It may be occupied by the nginx container - b. It may be occupied by an external web server - 2. On certificate renewal, nginx is not reloaded - """ - config = tutor_config.load(context.root) - runner = compose.ScriptRunner(context.root, config, context.docker_compose) - if not config["ACTIVATE_HTTPS"]: - fmt.echo_info("HTTPS is not activated: certificate generation skipped") - return - - script = runner.render("hooks", "certbot", "create") - - if config["WEB_PROXY"]: - fmt.echo_info( - """You are running Tutor behind a web proxy (WEB_PROXY=true): SSL/TLS -certificates must be generated on the host. For instance, to generate -certificates with Let's Encrypt, run: - -{} - -See the official certbot documentation for your platform: https://certbot.eff.org/""".format( - indent(script, " ") - ) - ) - return - - utils.docker_run( - "--volume", - "{}:/etc/letsencrypt/".format(tutor_env.data_path(context.root, "letsencrypt")), - "-p", - "80:80", - "--entrypoint=sh", - "docker.io/certbot/certbot:latest", - "-e", - "-c", - script, - ) - - -@click.command(help="Renew https certificates", name="renew") -@click.pass_obj -def https_renew(context): - config = tutor_config.load(context.root) - if not config["ACTIVATE_HTTPS"]: - fmt.echo_info("HTTPS is not activated: certificate renewal skipped") - return - if config["WEB_PROXY"]: - fmt.echo_info( - """You are running Tutor behind a web proxy (WEB_PROXY=true): SSL/TLS -certificates must be renewed on the host. For instance, to renew Let's Encrypt -certificates, run: - - certbot renew - -See the official certbot documentation for your platform: https://certbot.eff.org/""" - ) - return - docker_run = [ - "--volume", - "{}:/etc/letsencrypt/".format(tutor_env.data_path(context.root, "letsencrypt")), - "-p", - "80:80", - "certbot/certbot:latest", - "renew", - ] - utils.docker_run(*docker_run) - - @click.command(help="Upgrade from a previous Open edX named release") @click.option( - "--from", "from_version", default="ironwood", type=click.Choice(["ironwood"]) + "--from", + "from_version", + default="juniper", + type=click.Choice(["ironwood", "juniper"]), ) @click.option("-I", "--non-interactive", is_flag=True, help="Run non-interactively") @click.pass_obj @@ -183,8 +104,14 @@ Are you sure you want to continue?""" fmt.question(question), default=True, abort=True, prompt_suffix=" " ) - if from_version == "ironwood": + running_version = from_version + if running_version == "ironwood": upgrade_from_ironwood(context, config) + running_version = "juniper" + + if running_version == "juniper": + upgrade_from_juniper(context, config) + running_version = "koa" def upgrade_from_ironwood(context, config): @@ -194,11 +121,11 @@ def upgrade_from_ironwood(context, config): click.echo(fmt.title("Stopping any existing platform")) compose.stop.callback([]) - if not config["ACTIVATE_MONGODB"]: + if not config["RUN_MONGODB"]: fmt.echo_info( - "You are not running MongDB (ACTIVATE_MONGODB=false). It is your " + "You are not running MongDB (RUN_MONGODB=false). It is your " "responsibility to upgrade your MongoDb instance to v3.6. There is " - "nothing left to do." + "nothing left to do to upgrade from Ironwood." ) return @@ -233,9 +160,37 @@ def upgrade_from_ironwood(context, config): compose.stop.callback([]) -https.add_command(https_create) -https.add_command(https_renew) -local.add_command(https) +def upgrade_from_juniper(context, config): + click.echo(fmt.title("Upgrading from Juniper")) + tutor_env.save(context.root, config) + + click.echo(fmt.title("Stopping any existing platform")) + compose.stop.callback([]) + + if not config["RUN_MYSQL"]: + fmt.echo_info( + "You are not running MySQL (RUN_MYSQL=false). It is your " + "responsibility to upgrade your MySQL instance to v5.7. There is " + "nothing left to do to upgrade from Juniper." + ) + return + + click.echo(fmt.title("Upgrading MySQL from v5.6 to v5.7")) + compose.start.callback(detach=True, services=["mysql"]) + compose.execute.callback( + [ + "mysql", + "bash", + "-e", + "-c", + "mysql_upgrade -u {} --password='{}'".format( + config["MYSQL_ROOT_USERNAME"], config["MYSQL_ROOT_PASSWORD"] + ), + ] + ) + compose.stop.callback([]) + + local.add_command(quickstart) local.add_command(upgrade) compose.add_commands(local) diff --git a/tutor/commands/plugins.py b/tutor/commands/plugins.py index f9d8775..4a356a7 100644 --- a/tutor/commands/plugins.py +++ b/tutor/commands/plugins.py @@ -50,11 +50,16 @@ def enable(context, plugin_names): ) -@click.command(help="Disable a plugin") +@click.command( + short_help="Disable a plugin", + help="Disable one or more plugins. Specify 'all' to disable all enabled plugins at once.", +) @click.argument("plugin_names", metavar="plugin", nargs=-1) @click.pass_obj def disable(context, plugin_names): config = tutor_config.load_user(context.root) + if "all" in plugin_names: + plugin_names = [plugin.name for plugin in plugins.iter_enabled(config)] for plugin_name in plugin_names: plugins.disable(config, plugin_name) delete_plugin(context.root, plugin_name) diff --git a/tutor/config.py b/tutor/config.py index d51120b..a0f98f0 100644 --- a/tutor/config.py +++ b/tutor/config.py @@ -134,7 +134,7 @@ def load_plugins(config, defaults): def is_service_activated(config, service): - return config["ACTIVATE_" + service.upper()] + return config["RUN_" + service.upper()] def upgrade_obsolete(config): @@ -147,16 +147,35 @@ def upgrade_obsolete(config): config["OPENEDX_MYSQL_DATABASE"] = config.pop("MYSQL_DATABASE") if "MYSQL_USERNAME" in config: config["OPENEDX_MYSQL_USERNAME"] = config.pop("MYSQL_USERNAME") - if "ACTIVATE_NOTES" in config: - if config["ACTIVATE_NOTES"]: + if "RUN_NOTES" in config: + if config["RUN_NOTES"]: plugins.enable(config, "notes") - config.pop("ACTIVATE_NOTES") - if "ACTIVATE_XQUEUE" in config: - if config["ACTIVATE_XQUEUE"]: + config.pop("RUN_NOTES") + if "RUN_XQUEUE" in config: + if config["RUN_XQUEUE"]: plugins.enable(config, "xqueue") - config.pop("ACTIVATE_XQUEUE") + config.pop("RUN_XQUEUE") if "SECRET_KEY" in config: config["OPENEDX_SECRET_KEY"] = config.pop("SECRET_KEY") + # Replace WEB_PROXY by RUN_CADDY + if "WEB_PROXY" in config: + config["RUN_CADDY"] = not config.pop("WEB_PROXY") + # Rename ACTIVATE_HTTPS to ENABLE_HTTPS + if "ACTIVATE_HTTPS" in config: + config["ENABLE_HTTPS"] = config.pop("ACTIVATE_HTTPS") + # Replace RUN_* variables by RUN_* + for name in [ + "ACTIVATE_LMS", + "ACTIVATE_CMS", + "ACTIVATE_FORUM", + "ACTIVATE_ELASTICSEARCH", + "ACTIVATE_MONGODB", + "ACTIVATE_MYSQL", + "ACTIVATE_REDIS", + "ACTIVATE_SMTP", + ]: + if name in config: + config[name.replace("ACTIVATE_", "RUN_")] = config.pop(name) def convert_json2yml(root): diff --git a/tutor/env.py b/tutor/env.py index d8cabec..d0382d6 100644 --- a/tutor/env.py +++ b/tutor/env.py @@ -318,7 +318,7 @@ def current_release(root): """ Return the name of the current Open edX release. """ - return {"0": "ironwood", "3": "ironwood", "10": "juniper"}[ + return {"0": "ironwood", "3": "ironwood", "10": "juniper", "11": "koa"}[ current_version(root).split(".")[0] ] diff --git a/tutor/interactive.py b/tutor/interactive.py index 1759f7e..e9ce268 100644 --- a/tutor/interactive.py +++ b/tutor/interactive.py @@ -40,7 +40,7 @@ def ask_questions(config, defaults): dev_values = { "LMS_HOST": "local.overhang.io", "CMS_HOST": "studio.local.overhang.io", - "ACTIVATE_HTTPS": False, + "ENABLE_HTTPS": False, } fmt.echo_info( """As you are not running this platform in production, we automatically set the following configuration values:""" @@ -150,7 +150,7 @@ def ask_questions(config, defaults): "Activate SSL/TLS certificates for HTTPS access? Important note:" " this will NOT work in a development environment." ), - "ACTIVATE_HTTPS", + "ENABLE_HTTPS", config, defaults, ) diff --git a/tutor/templates/android/tutor.yaml b/tutor/templates/android/tutor.yaml index af37f7f..d5ecc8d 100644 --- a/tutor/templates/android/tutor.yaml +++ b/tutor/templates/android/tutor.yaml @@ -1,5 +1,5 @@ # See docs: https://openedx.atlassian.net/wiki/spaces/LEARNER/pages/48792067/App+Configuration+Flags -API_HOST_URL: "{{ "https" if ACTIVATE_HTTPS else "http" }}://{{ LMS_HOST }}" +API_HOST_URL: "{{ "https" if ENABLE_HTTPS else "http" }}://{{ LMS_HOST }}" ENVIRONMENT_DISPLAY_NAME: "tutor" PLATFORM_NAME: "{{ PLATFORM_NAME }}" PLATFORM_DESTINATION_NAME: "{{ LMS_HOST }}" diff --git a/tutor/templates/apps/caddy/Caddyfile b/tutor/templates/apps/caddy/Caddyfile new file mode 100644 index 0000000..270f00f --- /dev/null +++ b/tutor/templates/apps/caddy/Caddyfile @@ -0,0 +1,11 @@ +{{ LMS_HOST }}{% if not ENABLE_HTTPS %}:80{% endif %} { + reverse_proxy nginx:80 +} +preview.{{ LMS_HOST }}{% if not ENABLE_HTTPS %}:80{% endif %} { + reverse_proxy nginx:80 +} +{{ CMS_HOST }}{% if not ENABLE_HTTPS %}:80{% endif %} { + reverse_proxy nginx:80 +} + +{{ patch("caddyfile") }} \ No newline at end of file diff --git a/tutor/templates/apps/nginx/cms.conf b/tutor/templates/apps/nginx/cms.conf index 0b8cfae..cbe909a 100644 --- a/tutor/templates/apps/nginx/cms.conf +++ b/tutor/templates/apps/nginx/cms.conf @@ -1,25 +1,12 @@ -{% if ACTIVATE_CMS %} +{% if RUN_CMS %} upstream cms-backend { server cms:8000 fail_timeout=0; } -{% if ACTIVATE_HTTPS %} server { - server_name {{ CMS_HOST }}; - listen 80; - return 301 https://$server_name$request_uri; -} -{% endif %} - -server { - {% if ACTIVATE_HTTPS %}listen 443 {{ "" if WEB_PROXY else "ssl" }};{% else %}listen 80;{% endif %} + listen 80; server_name {{ CMS_HOST }}; - {% if ACTIVATE_HTTPS and not WEB_PROXY %} - ssl_certificate /etc/letsencrypt/live/{{ LMS_HOST }}/fullchain.pem; - ssl_certificate_key /etc/letsencrypt/live/{{ LMS_HOST }}/privkey.pem; - {% endif %} - access_log /var/log/nginx/access.log tutor; client_max_body_size 250M; server_tokens off; @@ -27,14 +14,8 @@ server { rewrite ^(.*)/favicon.ico$ /static/images/favicon.ico last; location @proxy_to_cms_app { - {% if not WEB_PROXY %} - proxy_set_header X-Forwarded-Proto $scheme; - proxy_set_header X-Forwarded-Port $server_port; - proxy_set_header X-Forwarded-For $remote_addr; - {% endif %} - proxy_set_header Host $http_host; proxy_redirect off; - + proxy_set_header Host $http_host; proxy_pass http://cms-backend; } @@ -42,11 +23,6 @@ server { try_files $uri @proxy_to_cms_app; } - location ~ ^/media/(?P.*) { - root /var/www/openedx-media; - try_files /$file =404; - expires 31536000s; - } - {{ patch("nginx-cms") }} + {{ patch("nginx-cms")|indent(2) }} } {% endif %} diff --git a/tutor/templates/apps/nginx/lms.conf b/tutor/templates/apps/nginx/lms.conf index 5f2e193..6b0d135 100644 --- a/tutor/templates/apps/nginx/lms.conf +++ b/tutor/templates/apps/nginx/lms.conf @@ -1,26 +1,12 @@ -{% if ACTIVATE_LMS %} +{% if RUN_LMS %} upstream lms-backend { server lms:8000 fail_timeout=0; } -{% if ACTIVATE_HTTPS %} server { - server_name {{ LMS_HOST }} preview.{{ LMS_HOST }}; - listen 80; - access_log /var/log/nginx/access.log tutor; - return 301 https://$server_name$request_uri; -} -{% endif %} - -server { - {% if ACTIVATE_HTTPS %}listen 443 {{ "" if WEB_PROXY else "ssl" }};{% else %}listen 80;{% endif %} + listen 80; server_name {{ LMS_HOST }} preview.{{ LMS_HOST }}; - {% if ACTIVATE_HTTPS and not WEB_PROXY %} - ssl_certificate /etc/letsencrypt/live/{{ LMS_HOST }}/fullchain.pem; - ssl_certificate_key /etc/letsencrypt/live/{{ LMS_HOST }}/privkey.pem; - {% endif %} - access_log /var/log/nginx/access.log tutor; client_max_body_size 4M; server_tokens off; @@ -28,14 +14,8 @@ server { rewrite ^(.*)/favicon.ico$ /static/images/favicon.ico last; location @proxy_to_lms_app { - {% if not WEB_PROXY %} - proxy_set_header X-Forwarded-Proto $scheme; - proxy_set_header X-Forwarded-Port $server_port; - proxy_set_header X-Forwarded-For $remote_addr; - {% endif %} - proxy_set_header Host $http_host; proxy_redirect off; - + proxy_set_header Host $http_host; proxy_pass http://lms-backend; } @@ -59,11 +39,6 @@ server { client_max_body_size 1049576; } - location ~ ^/media/(?P.*) { - root /var/www/openedx-media; - try_files /$file =404; - expires 31536000s; - } - {{ patch("nginx-lms") }} + {{ patch("nginx-lms")|indent(2) }} } {% endif %} diff --git a/tutor/templates/apps/openedx/config/cms.env.json b/tutor/templates/apps/openedx/config/cms.env.json index 0cc4b9d..85543ee 100644 --- a/tutor/templates/apps/openedx/config/cms.env.json +++ b/tutor/templates/apps/openedx/config/cms.env.json @@ -15,15 +15,15 @@ "ENABLE_LEARNER_RECORDS": false, "ENABLE_LIBRARY_INDEX": true }, - "LMS_ROOT_URL": "{{ "https" if ACTIVATE_HTTPS else "http" }}://{{ LMS_HOST }}", - "CMS_ROOT_URL": "{{ "https" if ACTIVATE_HTTPS else "http" }}://{{ CMS_HOST }}", + "LMS_ROOT_URL": "{{ "https" if ENABLE_HTTPS else "http" }}://{{ LMS_HOST }}", + "CMS_ROOT_URL": "{{ "https" if ENABLE_HTTPS else "http" }}://{{ CMS_HOST }}", "CMS_BASE": "{{ CMS_HOST }}", "LMS_BASE": "{{ LMS_HOST }}", "CONTACT_EMAIL": "{{ CONTACT_EMAIL }}", - "CELERY_BROKER_TRANSPORT": "amqp", - "CELERY_BROKER_HOSTNAME": "{{ RABBITMQ_HOST }}", - "CELERY_BROKER_USER": "{{ RABBITMQ_USERNAME }}", - "CELERY_BROKER_PASSWORD": "{{ RABBITMQ_PASSWORD }}", + "CELERY_BROKER_TRANSPORT": "redis", + "CELERY_BROKER_HOSTNAME": "{{ REDIS_HOST }}:{{ REDIS_PORT }}", + "CELERY_BROKER_USER": "{{ REDIS_USERNAME }}", + "CELERY_BROKER_PASSWORD": "{{ REDIS_PASSWORD }}", "ALTERNATE_WORKER_QUEUES": "lms", "ENABLE_COMPREHENSIVE_THEMING": true, "COMPREHENSIVE_THEME_DIRS": ["/openedx/themes"], @@ -37,7 +37,7 @@ "EMAIL_HOST": "{{ SMTP_HOST }}", "EMAIL_PORT": {{ SMTP_PORT }}, "EMAIL_USE_TLS": {{ "true" if SMTP_USE_TLS else "false" }}, - "HTTPS": "{{ "on" if ACTIVATE_HTTPS else "off" }}", + "HTTPS": "{{ "on" if ENABLE_HTTPS else "off" }}", "LANGUAGE_CODE": "{{ LANGUAGE_CODE }}", "SESSION_COOKIE_DOMAIN": ".{{ LMS_HOST|common_domain(CMS_HOST) }}", {{ patch("cms-env", separator=",\n", suffix=",")|indent(2) }} @@ -45,48 +45,41 @@ "default": { "KEY_PREFIX": "default", "VERSION": "1", - "BACKEND": "django.core.cache.backends.memcached.MemcachedCache", - "KEY_FUNCTION": "util.memcache.safe_key", - "LOCATION": "{{ MEMCACHED_HOST }}:{{ MEMCACHED_PORT }}" + "BACKEND": "django_redis.cache.RedisCache", + "LOCATION": "redis://{% if REDIS_USERNAME and REDIS_PASSWORD %}{{ REDIS_USERNAME }}:{{ REDIS_PASSWORD }}{% endif %}@{{ REDIS_HOST }}:{{ REDIS_PORT }}/1" }, "general": { "KEY_PREFIX": "general", - "BACKEND": "django.core.cache.backends.memcached.MemcachedCache", - "KEY_FUNCTION": "util.memcache.safe_key", - "LOCATION": "{{ MEMCACHED_HOST }}:{{ MEMCACHED_PORT }}" + "BACKEND": "django_redis.cache.RedisCache", + "LOCATION": "redis://{% if REDIS_USERNAME and REDIS_PASSWORD %}{{ REDIS_USERNAME }}:{{ REDIS_PASSWORD }}{% endif %}@{{ REDIS_HOST }}:{{ REDIS_PORT }}/1" }, "mongo_metadata_inheritance": { "KEY_PREFIX": "mongo_metadata_inheritance", "TIMEOUT": 300, - "BACKEND": "django.core.cache.backends.memcached.MemcachedCache", - "KEY_FUNCTION": "util.memcache.safe_key", - "LOCATION": "{{ MEMCACHED_HOST }}:{{ MEMCACHED_PORT }}" + "BACKEND": "django_redis.cache.RedisCache", + "LOCATION": "redis://{% if REDIS_USERNAME and REDIS_PASSWORD %}{{ REDIS_USERNAME }}:{{ REDIS_PASSWORD }}{% endif %}@{{ REDIS_HOST }}:{{ REDIS_PORT }}/1" }, "staticfiles": { "KEY_PREFIX": "staticfiles_cms", - "BACKEND": "django.core.cache.backends.memcached.MemcachedCache", - "KEY_FUNCTION": "util.memcache.safe_key", - "LOCATION": "{{ MEMCACHED_HOST }}:{{ MEMCACHED_PORT }}" + "BACKEND": "django.core.cache.backends.locmem.LocMemCache", + "LOCATION": "staticfiles_cms" }, "configuration": { "KEY_PREFIX": "configuration", - "BACKEND": "django.core.cache.backends.memcached.MemcachedCache", - "KEY_FUNCTION": "util.memcache.safe_key", - "LOCATION": "{{ MEMCACHED_HOST }}:{{ MEMCACHED_PORT }}" + "BACKEND": "django_redis.cache.RedisCache", + "LOCATION": "redis://{% if REDIS_USERNAME and REDIS_PASSWORD %}{{ REDIS_USERNAME }}:{{ REDIS_PASSWORD }}{% endif %}@{{ REDIS_HOST }}:{{ REDIS_PORT }}/1" }, "celery": { "KEY_PREFIX": "celery", "TIMEOUT": "7200", - "BACKEND": "django.core.cache.backends.memcached.MemcachedCache", - "KEY_FUNCTION": "util.memcache.safe_key", - "LOCATION": "{{ MEMCACHED_HOST }}:{{ MEMCACHED_PORT }}" + "BACKEND": "django_redis.cache.RedisCache", + "LOCATION": "redis://{% if REDIS_USERNAME and REDIS_PASSWORD %}{{ REDIS_USERNAME }}:{{ REDIS_PASSWORD }}{% endif %}@{{ REDIS_HOST }}:{{ REDIS_PORT }}/1" }, "course_structure_cache": { "KEY_PREFIX": "course_structure", "TIMEOUT": "7200", - "BACKEND": "django.core.cache.backends.memcached.MemcachedCache", - "KEY_FUNCTION": "util.memcache.safe_key", - "LOCATION": "{{ MEMCACHED_HOST }}:{{ MEMCACHED_PORT }}" + "BACKEND": "django_redis.cache.RedisCache", + "LOCATION": "redis://{% if REDIS_USERNAME and REDIS_PASSWORD %}{{ REDIS_USERNAME }}:{{ REDIS_PASSWORD }}{% endif %}@{{ REDIS_HOST }}:{{ REDIS_PORT }}/1" } }, {% include "apps/openedx/config/partials/auth.json" %} diff --git a/tutor/templates/apps/openedx/config/lms.env.json b/tutor/templates/apps/openedx/config/lms.env.json index 4941a77..0d3707b 100644 --- a/tutor/templates/apps/openedx/config/lms.env.json +++ b/tutor/templates/apps/openedx/config/lms.env.json @@ -22,15 +22,15 @@ "ENABLE_OAUTH2_PROVIDER": true, "ENABLE_THIRD_PARTY_AUTH": true }, - "LMS_ROOT_URL": "{{ "https" if ACTIVATE_HTTPS else "http" }}://{{ LMS_HOST }}", - "CMS_ROOT_URL": "{{ "https" if ACTIVATE_HTTPS else "http" }}://{{ CMS_HOST }}", + "LMS_ROOT_URL": "{{ "https" if ENABLE_HTTPS else "http" }}://{{ LMS_HOST }}", + "CMS_ROOT_URL": "{{ "https" if ENABLE_HTTPS else "http" }}://{{ CMS_HOST }}", "CMS_BASE": "{{ CMS_HOST }}", "LMS_BASE": "{{ LMS_HOST }}", "CONTACT_EMAIL": "{{ CONTACT_EMAIL }}", - "CELERY_BROKER_TRANSPORT": "amqp", - "CELERY_BROKER_HOSTNAME": "{{ RABBITMQ_HOST }}", - "CELERY_BROKER_USER": "{{ RABBITMQ_USERNAME }}", - "CELERY_BROKER_PASSWORD": "{{ RABBITMQ_PASSWORD }}", + "CELERY_BROKER_TRANSPORT": "redis", + "CELERY_BROKER_HOSTNAME": "{{ REDIS_HOST }}:{{ REDIS_PORT }}", + "CELERY_BROKER_USER": "{{ REDIS_USERNAME }}", + "CELERY_BROKER_PASSWORD": "{{ REDIS_PASSWORD }}", "ALTERNATE_WORKER_QUEUES": "cms", "COMMENTS_SERVICE_URL": "http://{{ FORUM_HOST }}:4567", "COMMENTS_SERVICE_KEY": "forumapikey", @@ -46,7 +46,7 @@ "EMAIL_HOST": "{{ SMTP_HOST }}", "EMAIL_PORT": {{ SMTP_PORT }}, "EMAIL_USE_TLS": {{ "true" if SMTP_USE_TLS else "false" }}, - "HTTPS": "{{ "on" if ACTIVATE_HTTPS else "off" }}", + "HTTPS": "{{ "on" if ENABLE_HTTPS else "off" }}", "LANGUAGE_CODE": "{{ LANGUAGE_CODE }}", "SESSION_COOKIE_DOMAIN": ".{{ LMS_HOST|common_domain(CMS_HOST) }}", {{ patch("lms-env", separator=",\n", suffix=",")|indent(2) }} @@ -54,54 +54,46 @@ "default": { "KEY_PREFIX": "default", "VERSION": "1", - "BACKEND": "django.core.cache.backends.memcached.MemcachedCache", - "KEY_FUNCTION": "util.memcache.safe_key", - "LOCATION": "{{ MEMCACHED_HOST }}:{{ MEMCACHED_PORT }}" + "BACKEND": "django_redis.cache.RedisCache", + "LOCATION": "redis://{% if REDIS_USERNAME and REDIS_PASSWORD %}{{ REDIS_USERNAME }}:{{ REDIS_PASSWORD }}{% endif %}@{{ REDIS_HOST }}:{{ REDIS_PORT }}/1" }, "general": { "KEY_PREFIX": "general", - "BACKEND": "django.core.cache.backends.memcached.MemcachedCache", - "KEY_FUNCTION": "util.memcache.safe_key", - "LOCATION": "{{ MEMCACHED_HOST }}:{{ MEMCACHED_PORT }}" + "BACKEND": "django_redis.cache.RedisCache", + "LOCATION": "redis://{% if REDIS_USERNAME and REDIS_PASSWORD %}{{ REDIS_USERNAME }}:{{ REDIS_PASSWORD }}{% endif %}@{{ REDIS_HOST }}:{{ REDIS_PORT }}/1" }, "mongo_metadata_inheritance": { "KEY_PREFIX": "mongo_metadata_inheritance", "TIMEOUT": 300, - "BACKEND": "django.core.cache.backends.memcached.MemcachedCache", - "KEY_FUNCTION": "util.memcache.safe_key", - "LOCATION": "{{ MEMCACHED_HOST }}:{{ MEMCACHED_PORT }}" + "BACKEND": "django_redis.cache.RedisCache", + "LOCATION": "redis://{% if REDIS_USERNAME and REDIS_PASSWORD %}{{ REDIS_USERNAME }}:{{ REDIS_PASSWORD }}{% endif %}@{{ REDIS_HOST }}:{{ REDIS_PORT }}/1" }, "staticfiles": { "KEY_PREFIX": "staticfiles_lms", - "BACKEND": "django.core.cache.backends.memcached.MemcachedCache", - "KEY_FUNCTION": "util.memcache.safe_key", - "LOCATION": "{{ MEMCACHED_HOST }}:{{ MEMCACHED_PORT }}" + "BACKEND": "django.core.cache.backends.locmem.LocMemCache", + "LOCATION": "staticfiles_lms" }, "configuration": { "KEY_PREFIX": "configuration", - "BACKEND": "django.core.cache.backends.memcached.MemcachedCache", - "KEY_FUNCTION": "util.memcache.safe_key", - "LOCATION": "{{ MEMCACHED_HOST }}:{{ MEMCACHED_PORT }}" + "BACKEND": "django_redis.cache.RedisCache", + "LOCATION": "redis://{% if REDIS_USERNAME and REDIS_PASSWORD %}{{ REDIS_USERNAME }}:{{ REDIS_PASSWORD }}{% endif %}@{{ REDIS_HOST }}:{{ REDIS_PORT }}/1" }, "celery": { "KEY_PREFIX": "celery", "TIMEOUT": "7200", - "BACKEND": "django.core.cache.backends.memcached.MemcachedCache", - "KEY_FUNCTION": "util.memcache.safe_key", - "LOCATION": "{{ MEMCACHED_HOST }}:{{ MEMCACHED_PORT }}" + "BACKEND": "django_redis.cache.RedisCache", + "LOCATION": "redis://{% if REDIS_USERNAME and REDIS_PASSWORD %}{{ REDIS_USERNAME }}:{{ REDIS_PASSWORD }}{% endif %}@{{ REDIS_HOST }}:{{ REDIS_PORT }}/1" }, "course_structure_cache": { "KEY_PREFIX": "course_structure", "TIMEOUT": "7200", - "BACKEND": "django.core.cache.backends.memcached.MemcachedCache", - "KEY_FUNCTION": "util.memcache.safe_key", - "LOCATION": "{{ MEMCACHED_HOST }}:{{ MEMCACHED_PORT }}" + "BACKEND": "django_redis.cache.RedisCache", + "LOCATION": "redis://{% if REDIS_USERNAME and REDIS_PASSWORD %}{{ REDIS_USERNAME }}:{{ REDIS_PASSWORD }}{% endif %}@{{ REDIS_HOST }}:{{ REDIS_PORT }}/1" }, "ora2-storage": { "KEY_PREFIX": "ora2-storage", - "BACKEND": "django.core.cache.backends.memcached.MemcachedCache", - "KEY_FUNCTION": "util.memcache.safe_key", - "LOCATION": "{{ MEMCACHED_HOST }}:{{ MEMCACHED_PORT }}" + "BACKEND": "django_redis.cache.RedisCache", + "LOCATION": "redis://{% if REDIS_USERNAME and REDIS_PASSWORD %}{{ REDIS_USERNAME }}:{{ REDIS_PASSWORD }}{% endif %}@{{ REDIS_HOST }}:{{ REDIS_PORT }}/1" } }, {% include "apps/openedx/config/partials/auth.json" %} diff --git a/tutor/templates/apps/openedx/settings/cms/__init__.py b/tutor/templates/apps/openedx/settings/cms/__init__.py index e69de29..f0c04d9 100644 --- a/tutor/templates/apps/openedx/settings/cms/__init__.py +++ b/tutor/templates/apps/openedx/settings/cms/__init__.py @@ -0,0 +1 @@ +{% include "apps/openedx/settings/partials/pre_common_all.py" %} diff --git a/tutor/templates/apps/openedx/settings/cms/production.py b/tutor/templates/apps/openedx/settings/cms/production.py index 1549e32..6b04a88 100644 --- a/tutor/templates/apps/openedx/settings/cms/production.py +++ b/tutor/templates/apps/openedx/settings/cms/production.py @@ -9,6 +9,4 @@ ALLOWED_HOSTS = [ "cms", ] -MIDDLEWARE.insert(0, "whitenoise.middleware.WhiteNoiseMiddleware") - {{ patch("openedx-cms-production-settings") }} diff --git a/tutor/templates/apps/openedx/settings/lms/__init__.py b/tutor/templates/apps/openedx/settings/lms/__init__.py index e69de29..f0c04d9 100644 --- a/tutor/templates/apps/openedx/settings/lms/__init__.py +++ b/tutor/templates/apps/openedx/settings/lms/__init__.py @@ -0,0 +1 @@ +{% include "apps/openedx/settings/partials/pre_common_all.py" %} diff --git a/tutor/templates/apps/openedx/settings/lms/production.py b/tutor/templates/apps/openedx/settings/lms/production.py index bc9293c..fad463c 100644 --- a/tutor/templates/apps/openedx/settings/lms/production.py +++ b/tutor/templates/apps/openedx/settings/lms/production.py @@ -10,9 +10,7 @@ ALLOWED_HOSTS = [ "lms", ] -MIDDLEWARE.insert(0, "whitenoise.middleware.WhiteNoiseMiddleware") - -{% if ACTIVATE_HTTPS %} +{% if ENABLE_HTTPS %} # Properly set the "secure" attribute on session/csrf cookies. This is required in # Chrome to support samesite=none cookies. SESSION_COOKIE_SECURE = True diff --git a/tutor/templates/apps/openedx/settings/partials/common_all.py b/tutor/templates/apps/openedx/settings/partials/common_all.py index 3cb5731..b46de53 100644 --- a/tutor/templates/apps/openedx/settings/partials/common_all.py +++ b/tutor/templates/apps/openedx/settings/partials/common_all.py @@ -2,6 +2,8 @@ import json import os +from xmodule.modulestore.modulestore_settings import update_module_store_settings + # Mongodb connection parameters: simply modify `mongodb_parameters` to affect all connections to MongoDb. mongodb_parameters = { "host": "{{ MONGODB_HOST }}", @@ -27,6 +29,8 @@ DATA_DIR = "/openedx/data/" for store in MODULESTORE["default"]["OPTIONS"]["stores"]: store["OPTIONS"]["fs_root"] = DATA_DIR +# Behave like memcache when it comes to connection errors +DJANGO_REDIS_IGNORE_EXCEPTIONS = True DEFAULT_FROM_EMAIL = ENV_TOKENS.get("DEFAULT_FROM_EMAIL", ENV_TOKENS["CONTACT_EMAIL"]) DEFAULT_FEEDBACK_EMAIL = ENV_TOKENS.get("DEFAULT_FEEDBACK_EMAIL", ENV_TOKENS["CONTACT_EMAIL"]) @@ -42,7 +46,7 @@ API_ACCESS_MANAGER_EMAIL = ENV_TOKENS.get("API_ACCESS_MANAGER_EMAIL", ENV_TOKENS API_ACCESS_FROM_EMAIL = ENV_TOKENS.get("API_ACCESS_FROM_EMAIL", ENV_TOKENS["CONTACT_EMAIL"]) # Get rid completely of coursewarehistoryextended, as we do not use the CSMH database -INSTALLED_APPS.remove("coursewarehistoryextended") +INSTALLED_APPS.remove("lms.djangoapps.coursewarehistoryextended") DATABASE_ROUTERS.remove( "openedx.core.lib.django_courseware_routers.StudentModuleHistoryExtendedRouter" ) @@ -82,15 +86,6 @@ LOGGING["handlers"]["tracking"] = { "formatter": "standard", } LOGGING["loggers"]["tracking"]["handlers"] = ["console", "local", "tracking"] -# Disable django/drf deprecation warnings -import logging -import warnings -from django.utils.deprecation import RemovedInDjango30Warning, RemovedInDjango31Warning -from rest_framework import RemovedInDRF310Warning, RemovedInDRF311Warning -warnings.simplefilter('ignore', RemovedInDjango30Warning) -warnings.simplefilter('ignore', RemovedInDjango31Warning) -warnings.simplefilter('ignore', RemovedInDRF310Warning) -warnings.simplefilter('ignore', RemovedInDRF311Warning) # Email EMAIL_USE_SSL = {{ SMTP_USE_SSL }} diff --git a/tutor/templates/apps/openedx/settings/partials/common_lms.py b/tutor/templates/apps/openedx/settings/partials/common_lms.py index 5eb709d..ef9bd6b 100644 --- a/tutor/templates/apps/openedx/settings/partials/common_lms.py +++ b/tutor/templates/apps/openedx/settings/partials/common_lms.py @@ -3,6 +3,10 @@ ######## Common LMS settings LOGIN_REDIRECT_WHITELIST = ["{{ CMS_HOST }}"] +# Better layout of honor code/tos links during registration +REGISTRATION_EXTRA_FIELDS["terms_of_service"] = "required" +REGISTRATION_EXTRA_FIELDS["honor_code"] = "hidden" + # This url must not be None and should not be used anywhere LEARNING_MICROFRONTEND_URL = "http://learn.openedx.org" diff --git a/tutor/templates/apps/openedx/settings/partials/pre_common_all.py b/tutor/templates/apps/openedx/settings/partials/pre_common_all.py new file mode 100644 index 0000000..282db63 --- /dev/null +++ b/tutor/templates/apps/openedx/settings/partials/pre_common_all.py @@ -0,0 +1,10 @@ +# Silence overly verbose warnings +import logging +import warnings +from django.utils.deprecation import RemovedInDjango30Warning, RemovedInDjango31Warning +from rest_framework import RemovedInDRF310Warning, RemovedInDRF311Warning +warnings.simplefilter('ignore', RemovedInDjango30Warning) +warnings.simplefilter('ignore', RemovedInDjango31Warning) +warnings.simplefilter('ignore', RemovedInDRF310Warning) +warnings.simplefilter('ignore', RemovedInDRF311Warning) +warnings.simplefilter('ignore', DeprecationWarning) diff --git a/tutor/templates/apps/redis/redis.conf b/tutor/templates/apps/redis/redis.conf new file mode 100644 index 0000000..cfdf709 --- /dev/null +++ b/tutor/templates/apps/redis/redis.conf @@ -0,0 +1,41 @@ +# https://raw.githubusercontent.com/redis/redis/6.0/redis.conf +port 6379 + +tcp-backlog 511 +timeout 0 +tcp-keepalive 300 +daemonize no +supervised no +pidfile /var/run/redis_6379.pid +loglevel notice +logfile "" +databases 16 + +################################ SNAPSHOTTING ################################ +# +# Save the DB on disk: +# +# save + +save 900 1 +save 300 10 +save 60 10000 + +stop-writes-on-bgsave-error yes +rdbcompression yes +rdbchecksum yes +dir /openedx/redis/data/ +dbfilename dump.rdb +rdb-del-sync-files no + +############################## APPEND ONLY MODE ############################### + +# http://redis.io/topics/persistence +appendonly yes +appendfilename "appendonly.aof" +appendfsync everysec +no-appendfsync-on-rewrite no +auto-aof-rewrite-percentage 100 +auto-aof-rewrite-min-size 64mb +aof-load-truncated yes +aof-use-rdb-preamble yes diff --git a/tutor/templates/build/android/Dockerfile b/tutor/templates/build/android/Dockerfile index fc77f18..b586322 100644 --- a/tutor/templates/build/android/Dockerfile +++ b/tutor/templates/build/android/Dockerfile @@ -1,6 +1,7 @@ -FROM docker.io/ubuntu:18.04 +FROM docker.io/ubuntu:20.04 MAINTAINER Overhang.io +ENV DEBIAN_FRONTEND=noninteractive RUN apt update && \ apt upgrade -y && \ apt install -y wget unzip git openjdk-8-jre openjdk-8-jdk @@ -25,7 +26,7 @@ RUN yes | /openedx/android-sdk/tools/bin/sdkmanager --sdk_root=${ANDROID_HOME} - # Install android app repo ARG ANDROID_APP_REPOSITORY=https://github.com/edx/edx-app-android -ARG ANDROID_APP_VERSION=release/2.21.1 +ARG ANDROID_APP_VERSION=release/2.23.2 RUN git clone $ANDROID_APP_REPOSITORY --branch $ANDROID_APP_VERSION /openedx/edx-app-android WORKDIR /openedx/edx-app-android @@ -37,5 +38,3 @@ RUN ./gradlew tasks COPY ./edx.properties ./OpenEdXMobile/edx.properties RUN mkdir /openedx/config RUN ln -s /openedx/config/gradle.properties ./OpenEdXMobile/gradle.properties - -ENTRYPOINT ["sh", "-e", "-c"] \ No newline at end of file diff --git a/tutor/templates/build/forum/Dockerfile b/tutor/templates/build/forum/Dockerfile index c8ce370..33ac5e7 100644 --- a/tutor/templates/build/forum/Dockerfile +++ b/tutor/templates/build/forum/Dockerfile @@ -1,8 +1,7 @@ -FROM docker.io/ubuntu:18.04 +FROM docker.io/ubuntu:20.04 MAINTAINER Overhang.io -RUN mkdir /openedx - +ENV DEBIAN_FRONTEND=noninteractive RUN apt update && \ apt upgrade -y && \ apt install -y git wget autoconf bison build-essential libssl-dev libyaml-dev libreadline6-dev zlib1g-dev libncurses5-dev libffi-dev libgdbm-dev @@ -13,6 +12,8 @@ RUN wget -O /tmp/dockerize.tar.gz https://github.com/jwilder/dockerize/releases/ && tar -C /usr/local/bin -xzvf /tmp/dockerize.tar.gz \ && rm /tmp/dockerize.tar.gz +RUN mkdir /openedx + # Install ruby-build for building specific version of ruby # The ruby-build version should be periodically updated to reflect the latest release ARG RUBY_BUILD_VERSION=v20200401 @@ -39,7 +40,7 @@ RUN chmod a+x /openedx/bin/* ENV PATH /openedx/bin:${PATH} ENTRYPOINT ["docker-entrypoint.sh"] -ENV RACK_ENV staging +ENV SINATRA_ENV staging ENV NEW_RELIC_ENABLE false ENV API_KEY forumapikey ENV SEARCH_SERVER "http://elasticsearch:9200" diff --git a/tutor/templates/build/openedx-dev/Dockerfile b/tutor/templates/build/openedx-dev/Dockerfile index 8c04f9c..84bc198 100644 --- a/tutor/templates/build/openedx-dev/Dockerfile +++ b/tutor/templates/build/openedx-dev/Dockerfile @@ -8,8 +8,10 @@ RUN apt update && \ # Install dev python requirements RUN pip install -r requirements/edx/development.txt -# Pinning the jedi library is required in python 3.5 to avoid deprecation warnings from ipython -RUN pip install ipdb==0.13.2 ipython==7.9.0 jedi==0.15.2 +# We install ipython from source to avoid too many deprecation warnings +# https://github.com/ipython/ipython/issues/12206 +# We might be able to avoid this once they make a release later than 7.19.0. +RUN pip install ipdb==0.13.4 git+https://github.com/ipython/ipython.git@d0649a54a8936a8019d54549779dc92bcbde4e68#egg=ipython # Recompile static assets: in development mode all static assets are stored in edx-platform, # and the location of these files is stored in webpack-stats.json. If we don't recompile diff --git a/tutor/templates/build/openedx/Dockerfile b/tutor/templates/build/openedx/Dockerfile index 2cf8ed6..aec4288 100644 --- a/tutor/templates/build/openedx/Dockerfile +++ b/tutor/templates/build/openedx/Dockerfile @@ -1,7 +1,8 @@ ###### Minimal image with base system requirements for most stages -FROM docker.io/ubuntu:16.04 as minimal +FROM docker.io/ubuntu:20.04 as minimal MAINTAINER Overhang.io +ENV DEBIAN_FRONTEND=noninteractive RUN apt update && \ apt install -y build-essential curl git language-pack-en ENV LC_ALL en_US.UTF-8 @@ -13,11 +14,11 @@ RUN apt update && \ apt install -y libssl-dev zlib1g-dev libbz2-dev \ libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev libncursesw5-dev \ xz-utils tk-dev libffi-dev liblzma-dev python-openssl git -ARG PYTHON_VERSION=3.5.9 +ARG PYTHON_VERSION=3.8.6 ENV PYENV_ROOT /opt/pyenv -RUN git clone https://github.com/pyenv/pyenv $PYENV_ROOT --branch v1.2.18 --depth 1 +RUN git clone https://github.com/pyenv/pyenv $PYENV_ROOT --branch v1.2.21 --depth 1 RUN $PYENV_ROOT/bin/pyenv install $PYTHON_VERSION -RUN $PYENV_ROOT/versions/$PYTHON_VERSION/bin/pyvenv /openedx/venv +RUN $PYENV_ROOT/versions/$PYTHON_VERSION/bin/python -m venv /openedx/venv ###### Install Dockerize to wait for mysql DB availability FROM minimal as dockerize @@ -35,31 +36,19 @@ RUN mkdir -p /openedx/edx-platform && \ WORKDIR /openedx/edx-platform # Patch edx-platform -# Get rid of lepl-related warnings -# https://github.com/edx/edx-platform/pull/24059 -# https://github.com/overhangio/edx-platform/tree/overhangio/lepl-rfc6266-warning -RUN curl https://github.com/overhangio/edx-platform/commit/5f21bbe77056d71ca61b97b6badcff3c1a31b858.patch | git apply - -# Fix creation of LTI provider objects -# https://github.com/edx/edx-platform/pull/24055 -# https://github.com/overhangio/edx-platform/tree/regisb/fix-lti-provider-admin -RUN curl https://github.com/overhangio/edx-platform/commit/089b26eed0302ed1f9a5b24c5f3e563dd44abb04.patch | git apply - -# Fix problem viewing when CSMH is disabled -# https://github.com/edx/edx-platform/pull/24237 -# https://github.com/overhangio/edx-platform/tree/overhangio/fix-no-csmh -RUN curl https://github.com/overhangio/edx-platform/commit/6dbf2eddf7a4563c04c3b51edf5e131106d69e19.patch | git apply - -# Fix upload of video transcripts to s3 -# https://github.com/edx/edx-platform/pull/24800 -RUN curl https://github.com/edx/edx-platform/commit/80fa2cae128e2a1fd8ab298351b7b36c9d139e6c.patch | git apply - # Make it possible to disable learner records globally # https://github.com/edx/edx-platform/pull/25182 # https://github.com/overhangio/edx-platform/tree/overhangio/disable-learner-records-from-settings -RUN curl https://github.com/overhangio/edx-platform/commit/58f20a0547355080eeee346104a1719ad806902e.patch | git apply - -# Fix security issues -# https://github.com/edx/edx-platform/pull/25782/commits -RUN curl https://github.com/edx/edx-platform/commit/c03857b78d6204ed3b9a3093367348ebfaaf7d04.patch | git apply - +RUN curl https://github.com/overhangio/edx-platform/commit/63e8fcb403f18ff24ef9ef6dbd5d38a7fa82fed6.patch | git apply - +# Fix inconvenient pavelib warning +# https://github.com/edx/edx-platform/pull/25771 +# https://github.com/overhangio/edx-platform/tree/overhangio/fix-paver-warning +RUN curl https://github.com/overhangio/edx-platform/commit/bc0ab09f9945bd14aa6be1dbbf928cce58f079d2.patch | git apply - ###### Download extra locales to /openedx/locale/contrib/locale FROM minimal as locales +# TODO: openedx-i18n is not yet tagged for koa.1: replace versions below by OPENEDX_COMMON_VERSION +ARG OPENEDX_I18N_VERSION=open-release/koa.test02 RUN cd /tmp \ && curl -L -o openedx-i18n.tar.gz https://github.com/openedx/openedx-i18n/archive/{{ OPENEDX_COMMON_VERSION }}.tar.gz \ && tar xzf /tmp/openedx-i18n.tar.gz \ @@ -86,14 +75,14 @@ RUN pip install setuptools==44.1.0 pip==20.0.2 wheel==0.34.2 # Install base requirements RUN pip install -r ./requirements/edx/base.txt -# Install patched version of ora2 -RUN pip install https://github.com/overhangio/edx-ora2/archive/overhangio/boto2to3.zip - # Install scorm xblock -RUN pip install "openedx-scorm-xblock<11.0.0,>=10.0.1" +RUN pip install "openedx-scorm-xblock<12.0.0,>=11.0.0" -# Install whitenoise, for serving static assets -RUN pip install "whitenoise==5.1.0" +# Install django-redis for using redis as a django cache +RUN pip install django-redis==4.12.1 + +# Install uwsgi +RUN pip install uwsgi==2.0.19.1 # Install private requirements: this is useful for installing custom xblocks. COPY ./requirements/ /openedx/requirements @@ -120,7 +109,7 @@ FROM minimal as production # Install system requirements RUN apt update && \ - apt install -y gettext gfortran graphviz graphviz-dev libffi-dev libfreetype6-dev libgeos-dev libjpeg8-dev liblapack-dev libmysqlclient-dev libpng12-dev libsqlite3-dev libxmlsec1-dev lynx ntp pkg-config rdfind && \ + apt install -y gettext gfortran graphviz graphviz-dev libffi-dev libfreetype6-dev libgeos-dev libjpeg8-dev liblapack-dev libmysqlclient-dev libpng-dev libsqlite3-dev libxmlsec1-dev lynx ntp pkg-config rdfind && \ rm -rf /var/lib/apt/lists/* COPY --from=dockerize /usr/local/bin/dockerize /usr/local/bin/dockerize @@ -203,6 +192,13 @@ ENV SETTINGS tutor.production ENTRYPOINT ["docker-entrypoint.sh"] # Run server -COPY gunicorn_conf.py /openedx/gunicorn_conf.py EXPOSE 8000 -CMD gunicorn -c /openedx/gunicorn_conf.py --name ${SERVICE_VARIANT} --bind=0.0.0.0:8000 --max-requests=1000 --max-requests-jitter=100 --timeout=120 --access-logfile - ${SERVICE_VARIANT}.wsgi:application +CMD uwsgi \ + --static-map /static=/openedx/staticfiles/ \ + --static-map /media=/openedx/media/ \ + --http 0.0.0.0:8000 \ + --thunder-lock \ + --single-interpreter \ + --enable-threads \ + --processes=${UWSGI_WORKERS:-2} \ + --wsgi-file ${SERVICE_VARIANT}/wsgi.py diff --git a/tutor/templates/build/openedx/bin/reload-gunicorn b/tutor/templates/build/openedx/bin/reload-gunicorn deleted file mode 100755 index 0b1b386..0000000 --- a/tutor/templates/build/openedx/bin/reload-gunicorn +++ /dev/null @@ -1,3 +0,0 @@ -#! /bin/bash -echo "Reloading gunicorn process..." -kill -HUP $(pidof /openedx/venv/bin/python3 /openedx/venv/bin/gunicorn) diff --git a/tutor/templates/build/openedx/bin/reload-uwsgi b/tutor/templates/build/openedx/bin/reload-uwsgi new file mode 100755 index 0000000..9be2665 --- /dev/null +++ b/tutor/templates/build/openedx/bin/reload-uwsgi @@ -0,0 +1,3 @@ +#! /bin/bash +echo "Reloading uwsgi process..." +kill -HUP $(pidof /openedx/venv/bin/python3 /openedx/venv/bin/uwsgi) diff --git a/tutor/templates/build/openedx/gunicorn_conf.py b/tutor/templates/build/openedx/gunicorn_conf.py deleted file mode 100644 index 40e6288..0000000 --- a/tutor/templates/build/openedx/gunicorn_conf.py +++ /dev/null @@ -1,4 +0,0 @@ -import os - -# Set the number of gunicorn workers -workers = int(os.environ.get("GUNICORN_WORKERS", "2")) \ No newline at end of file diff --git a/tutor/templates/build/openedx/revisions.yml b/tutor/templates/build/openedx/revisions.yml index ad9a269..a62b769 100644 --- a/tutor/templates/build/openedx/revisions.yml +++ b/tutor/templates/build/openedx/revisions.yml @@ -1 +1 @@ -EDX_PLATFORM_REVISION: juniper \ No newline at end of file +EDX_PLATFORM_REVISION: koa \ No newline at end of file diff --git a/tutor/templates/config.yml b/tutor/templates/config.yml index 786727e..7e08d2c 100644 --- a/tutor/templates/config.yml +++ b/tutor/templates/config.yml @@ -10,16 +10,16 @@ ID: "{{ 24|random_string }}" LMS_HOST: "www.myopenedx.com" # The following are default values -ACTIVATE_LMS: true -ACTIVATE_CMS: true -ACTIVATE_FORUM: true -ACTIVATE_ELASTICSEARCH: true -ACTIVATE_HTTPS: false -ACTIVATE_MEMCACHED: true -ACTIVATE_MONGODB: true -ACTIVATE_MYSQL: true -ACTIVATE_RABBITMQ: true -ACTIVATE_SMTP: true +RUN_CADDY: true +RUN_LMS: true +RUN_CMS: true +RUN_FORUM: true +RUN_ELASTICSEARCH: true +ENABLE_HTTPS: false +RUN_MONGODB: true +RUN_MYSQL: true +RUN_REDIS: true +RUN_SMTP: true CMS_HOST: "studio.{{ LMS_HOST }}" CONTACT_EMAIL: "contact@{{ LMS_HOST }}" OPENEDX_AWS_ACCESS_KEY: "" @@ -32,13 +32,13 @@ DOCKER_REGISTRY: "docker.io/" DOCKER_IMAGE_OPENEDX: "{{ DOCKER_REGISTRY }}overhangio/openedx:{{ TUTOR_VERSION }}" DOCKER_IMAGE_OPENEDX_DEV: "{{ DOCKER_REGISTRY }}overhangio/openedx-dev:{{ TUTOR_VERSION }}" DOCKER_IMAGE_ANDROID: "{{ DOCKER_REGISTRY }}overhangio/openedx-android:{{ TUTOR_VERSION }}" +DOCKER_IMAGE_CADDY: "{{ DOCKER_REGISTRY }}caddy:2.2.1" DOCKER_IMAGE_FORUM: "{{ DOCKER_REGISTRY }}overhangio/openedx-forum:{{ TUTOR_VERSION }}" -DOCKER_IMAGE_MEMCACHED: "{{ DOCKER_REGISTRY }}memcached:1.4.38" DOCKER_IMAGE_MONGODB: "{{ DOCKER_REGISTRY }}mongo:3.6.18" -DOCKER_IMAGE_MYSQL: "{{ DOCKER_REGISTRY }}mysql:5.6.49" +DOCKER_IMAGE_MYSQL: "{{ DOCKER_REGISTRY }}mysql:5.7.32" DOCKER_IMAGE_ELASTICSEARCH: "{{ DOCKER_REGISTRY }}elasticsearch:1.5.2" DOCKER_IMAGE_NGINX: "{{ DOCKER_REGISTRY }}nginx:1.13" -DOCKER_IMAGE_RABBITMQ: "{{ DOCKER_REGISTRY }}rabbitmq:3.6.10-management-alpine" +DOCKER_IMAGE_REDIS: "{{ DOCKER_REGISTRY }}redis:6.0.9" DOCKER_IMAGE_SMTP: "{{ DOCKER_REGISTRY }}namshi/smtp:latest" LOCAL_PROJECT_NAME: "tutor_local" ELASTICSEARCH_HOST: "elasticsearch" @@ -47,38 +47,35 @@ ELASTICSEARCH_SCHEME: "http" ELASTICSEARCH_HEAP_SIZE: 1g FORUM_HOST: "forum" JWT_COMMON_AUDIENCE: "openedx" -JWT_COMMON_ISSUER: "{% if ACTIVATE_HTTPS %}https{% else %}http{% endif %}://{{ LMS_HOST }}/oauth2" +JWT_COMMON_ISSUER: "{% if ENABLE_HTTPS %}https{% else %}http{% endif %}://{{ LMS_HOST }}/oauth2" JWT_COMMON_SECRET_KEY: "{{ OPENEDX_SECRET_KEY }}" JWT_RSA_PRIVATE_KEY: "{{ 2048|rsa_private_key }}" K8S_NAMESPACE: "openedx" LANGUAGE_CODE: "en" -MEMCACHED_HOST: "memcached" -MEMCACHED_PORT: 11211 MONGODB_HOST: "mongodb" MONGODB_DATABASE: "openedx" MONGODB_PORT: 27017 MONGODB_USERNAME: "" MONGODB_PASSWORD: "" -OPENEDX_CMS_GUNICORN_WORKERS: 2 -OPENEDX_LMS_GUNICORN_WORKERS: 2 +OPENEDX_CMS_UWSGI_WORKERS: 2 +OPENEDX_LMS_UWSGI_WORKERS: 2 OPENEDX_MYSQL_DATABASE: "openedx" OPENEDX_CSMH_MYSQL_DATABASE: "{{ OPENEDX_MYSQL_DATABASE }}_csmh" OPENEDX_MYSQL_USERNAME: "openedx" -OPENEDX_COMMON_VERSION: "open-release/juniper.3" +OPENEDX_COMMON_VERSION: "open-release/koa.1" MYSQL_HOST: "mysql" MYSQL_PORT: 3306 MYSQL_ROOT_USERNAME: "root" NGINX_HTTP_PORT: 80 -NGINX_HTTPS_PORT: 443 PLATFORM_NAME: "My Open edX" PLUGINS: [] -RABBITMQ_HOST: "rabbitmq" -RABBITMQ_USERNAME: "" -RABBITMQ_PASSWORD: "" +REDIS_HOST: "redis" +REDIS_PORT: 6379 +REDIS_USERNAME: "" +REDIS_PASSWORD: "" SMTP_HOST: "smtp" SMTP_PORT: 25 SMTP_USERNAME: "" SMTP_PASSWORD: "" SMTP_USE_TLS: false SMTP_USE_SSL: false -WEB_PROXY: false diff --git a/tutor/templates/hooks/certbot/create b/tutor/templates/hooks/certbot/create deleted file mode 100644 index 45c346e..0000000 --- a/tutor/templates/hooks/certbot/create +++ /dev/null @@ -1,2 +0,0 @@ -certbot certonly --standalone -n --agree-tos -m admin@{{ LMS_HOST }} -d {{ LMS_HOST }} -d {{ CMS_HOST }} -d preview.{{ LMS_HOST }} -{{ patch("https-create") }} \ No newline at end of file diff --git a/tutor/templates/hooks/forum/init b/tutor/templates/hooks/forum/init index 9a1532e..2e9cf9f 100644 --- a/tutor/templates/hooks/forum/init +++ b/tutor/templates/hooks/forum/init @@ -1,4 +1,2 @@ -export MONGOHQ_URL="mongodb://$MONGODB_AUTH$MONGODB_HOST:$MONGODB_PORT/cs_comments_service" - bundle exec rake search:initialize bundle exec rake search:rebuild_index diff --git a/tutor/templates/k8s/deployments.yml b/tutor/templates/k8s/deployments.yml index 15e8072..34d5156 100644 --- a/tutor/templates/k8s/deployments.yml +++ b/tutor/templates/k8s/deployments.yml @@ -1,3 +1,40 @@ +{% if RUN_CADDY %} +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: caddy + labels: + app.kubernetes.io/name: caddy +spec: + selector: + matchLabels: + app.kubernetes.io/name: caddy + template: + metadata: + labels: + app.kubernetes.io/name: caddy + spec: + containers: + - name: caddy + image: {{ DOCKER_IMAGE_CADDY }} + volumeMounts: + - mountPath: /etc/caddy/ + name: config + - mountPath: /data/ + name: data + ports: + - containerPort: 80 + - containerPort: 443 + volumes: + - name: config + configMap: + name: caddy-config + - name: data + persistentVolumeClaim: + claimName: caddy +{% endif %} +{% if RUN_CMS %} --- apiVersion: apps/v1 kind: Deployment @@ -61,7 +98,7 @@ spec: containers: - name: cms-worker image: {{ DOCKER_IMAGE_OPENEDX }} - args: ["./manage.py", "cms", "celery", "worker", "--loglevel=info", "--hostname=edx.cms.core.default.%%h", "--maxtasksperchild", "100", "--exclude-queues=edx.lms.core.default"] + args: ["celery", "worker", "--app=cms.celery", "--loglevel=info", "--hostname=edx.cms.core.default.%%h", "--maxtasksperchild", "100", "--exclude-queues=edx.lms.core.default"] env: - name: SERVICE_VARIANT value: cms @@ -84,7 +121,8 @@ spec: - name: config configMap: name: openedx-config -{% if ACTIVATE_FORUM %} +{% endif %} +{% if RUN_FORUM %} --- apiVersion: apps/v1 kind: Deployment @@ -116,6 +154,7 @@ spec: - name: MONGODB_PORT value: "{{ MONGODB_PORT }}" {% endif %} +{% if RUN_LMS %} --- apiVersion: apps/v1 kind: Deployment @@ -176,7 +215,7 @@ spec: containers: - name: lms-worker image: {{ DOCKER_IMAGE_OPENEDX }} - args: ["./manage.py", "lms", "celery", "worker", "--loglevel=info", "--hostname=edx.lms.core.default.%%h", "--maxtasksperchild", "100", "--exclude-queues=edx.cms.core.default"] + args: ["celery", "worker", "--app=cms.celery", "--loglevel=info", "--hostname=edx.lms.core.default.%%h", "--maxtasksperchild", "100", "--exclude-queues=edx.cms.core.default"] env: - name: SERVICE_VARIANT value: lms @@ -199,7 +238,8 @@ spec: - name: config configMap: name: openedx-config -{% if ACTIVATE_ELASTICSEARCH %} +{% endif %} +{% if RUN_ELASTICSEARCH %} --- apiVersion: apps/v1 kind: Deployment @@ -238,30 +278,7 @@ spec: persistentVolumeClaim: claimName: elasticsearch {% endif %} -{% if ACTIVATE_MEMCACHED %} ---- -apiVersion: apps/v1 -kind: Deployment -metadata: - name: memcached - labels: - app.kubernetes.io/name: memcached -spec: - selector: - matchLabels: - app.kubernetes.io/name: memcached - template: - metadata: - labels: - app.kubernetes.io/name: memcached - spec: - containers: - - name: memcached - image: {{ DOCKER_IMAGE_MEMCACHED }} - ports: - - containerPort: 11211 -{% endif %} -{% if ACTIVATE_MONGODB %} +{% if RUN_MONGODB %} --- apiVersion: apps/v1 kind: Deployment @@ -295,7 +312,7 @@ spec: persistentVolumeClaim: claimName: mongodb {% endif %} -{% if ACTIVATE_MYSQL %} +{% if RUN_MYSQL %} --- apiVersion: apps/v1 kind: Deployment @@ -317,7 +334,9 @@ spec: containers: - name: mysql image: {{ DOCKER_IMAGE_MYSQL }} - args: ["mysqld", "--character-set-server=utf8", "--collation-server=utf8_general_ci"] + # Note the ignore-db-dir: this is because ext4 volumes are created with a lost+found directory in them, which causes mysql + # initialization to fail + args: ["mysqld", "--character-set-server=utf8", "--collation-server=utf8_general_ci", "--ignore-db-dir=lost+found"] env: - name: MYSQL_ROOT_PASSWORD value: "{{ MYSQL_ROOT_PASSWORD }}" @@ -331,7 +350,7 @@ spec: persistentVolumeClaim: claimName: mysql {% endif %} -{% if ACTIVATE_SMTP %} +{% if RUN_SMTP %} --- apiVersion: apps/v1 kind: Deployment @@ -379,42 +398,46 @@ spec: {{ patch("k8s-deployments-nginx-volume-mounts")|indent(12) }} ports: - containerPort: 80 - - containerPort: 443 volumes: - name: config configMap: name: nginx-config {{ patch("k8s-deployments-nginx-volumes")|indent(8) }} -{% if ACTIVATE_RABBITMQ %} +{% if RUN_REDIS %} --- apiVersion: apps/v1 kind: Deployment metadata: - name: rabbitmq + name: redis labels: - app.kubernetes.io/name: rabbitmq + app.kubernetes.io/name: redis spec: selector: matchLabels: - app.kubernetes.io/name: rabbitmq + app.kubernetes.io/name: redis strategy: type: Recreate template: metadata: labels: - app.kubernetes.io/name: rabbitmq + app.kubernetes.io/name: redis spec: containers: - - name: rabbitmq - image: {{ DOCKER_IMAGE_RABBITMQ }} + - name: redis + image: {{ DOCKER_IMAGE_REDIS }} ports: - - containerPort: 5672 + - containerPort: {{ REDIS_PORT }} volumeMounts: - - mountPath: /var/lib/rabbitmq + - mountPath: /openedx/redis/config/ + name: config + - mountPath: /openedx/redis/data name: data volumes: + - name: config + configMap: + name: redis-config - name: data persistentVolumeClaim: - claimName: rabbitmq + claimName: redis {% endif %} {{ patch("k8s-deployments") }} diff --git a/tutor/templates/k8s/ingress.yml b/tutor/templates/k8s/ingress.yml deleted file mode 100644 index e990f8a..0000000 --- a/tutor/templates/k8s/ingress.yml +++ /dev/null @@ -1,57 +0,0 @@ ----{% set hosts = [LMS_HOST, "preview." + LMS_HOST, CMS_HOST] %} -# This is an nginx-based Ingress object that relies on a letsencrypt Issuer for SSL -# termination. By default, this ingress and issuer are *not* deployed to the Kubernetes -# cluster when running "quickstart". This is because there exist many different -# ingress/issuer combinations and it should not be Tutor's job to choose which one you -# should use. -apiVersion: networking.k8s.io/v1beta1 -kind: Ingress -metadata: - name: web - labels: - app.kubernetes.io/name: web - app.kubernetes.io/component: ingress - annotations: - kubernetes.io/ingress.class: nginx - nginx.ingress.kubernetes.io/proxy-body-size: 1000m - {% if ACTIVATE_HTTPS%}kubernetes.io/tls-acme: "true" - cert-manager.io/issuer: letsencrypt{% endif %} -spec: - rules: - {% for host in hosts %} - - host: {{ host }} - http: - paths: - - backend: - serviceName: nginx - servicePort: {% if ACTIVATE_HTTPS %}443{% else %}80{% endif %}{% endfor %} - {{ patch("k8s-ingress-rules")|indent(2) }} - {% if ACTIVATE_HTTPS %} - tls: - - hosts: - {% for host in hosts %} - - {{ host }}{% endfor %} - {{ patch("k8s-ingress-tls-hosts")|indent(6) }} - secretName: letsencrypt - {%endif%} -{% if ACTIVATE_HTTPS %} ---- -apiVersion: cert-manager.io/v1alpha2 -kind: Issuer -metadata: - name: letsencrypt - labels: - app.kubernetes.io/name: letsencrypt - app.kubernetes.io/component: issuer -spec: - acme: - server: https://acme-v02.api.letsencrypt.org/directory - email: {{ CONTACT_EMAIL }} - privateKeySecretRef: - name: letsencrypt-privatekey - solvers: - - selector: {} - http01: - ingress: - class: nginx -{% endif %} diff --git a/tutor/templates/k8s/services.yml b/tutor/templates/k8s/services.yml index 6aded5e..f50da21 100644 --- a/tutor/templates/k8s/services.yml +++ b/tutor/templates/k8s/services.yml @@ -1,3 +1,20 @@ +{% if RUN_CADDY %} +--- +apiVersion: v1 +kind: Service +metadata: + name: caddy +spec: + type: LoadBalancer + ports: + - port: 80 + name: http + - port: 443 + name: https + selector: + app.kubernetes.io/name: caddy +{% endif %} +{% if RUN_CMS %} --- apiVersion: v1 kind: Service @@ -10,7 +27,8 @@ spec: protocol: TCP selector: app.kubernetes.io/name: cms -{% if ACTIVATE_FORUM %} +{% endif %} +{% if RUN_FORUM %} --- apiVersion: v1 kind: Service @@ -24,6 +42,7 @@ spec: selector: app.kubernetes.io/name: forum {% endif %} +{% if RUN_LMS %} --- apiVersion: v1 kind: Service @@ -36,7 +55,8 @@ spec: protocol: TCP selector: app.kubernetes.io/name: lms -{% if ACTIVATE_ELASTICSEARCH %} +{% endif %} +{% if RUN_ELASTICSEARCH %} --- apiVersion: v1 kind: Service @@ -50,21 +70,7 @@ spec: selector: app.kubernetes.io/name: elasticsearch {% endif %} -{% if ACTIVATE_MEMCACHED %} ---- -apiVersion: v1 -kind: Service -metadata: - name: memcached -spec: - type: NodePort - ports: - - port: 11211 - protocol: TCP - selector: - app.kubernetes.io/name: memcached -{% endif %} -{% if ACTIVATE_MONGODB %} +{% if RUN_MONGODB %} --- apiVersion: v1 kind: Service @@ -78,7 +84,7 @@ spec: selector: app.kubernetes.io/name: mongodb {% endif %} -{% if ACTIVATE_MYSQL %} +{% if RUN_MYSQL %} --- apiVersion: v1 kind: Service @@ -102,25 +108,23 @@ spec: ports: - port: 80 name: http - - port: 443 - name: https selector: app.kubernetes.io/name: nginx -{% if ACTIVATE_RABBITMQ %} +{% if RUN_REDIS %} --- apiVersion: v1 kind: Service metadata: - name: rabbitmq + name: redis spec: type: NodePort ports: - - port: 5672 + - port: {{ REDIS_PORT }} protocol: TCP selector: - app.kubernetes.io/name: rabbitmq + app.kubernetes.io/name: redis {% endif %} -{% if ACTIVATE_SMTP %} +{% if RUN_SMTP %} --- apiVersion: v1 kind: Service diff --git a/tutor/templates/k8s/volumes.yml b/tutor/templates/k8s/volumes.yml index 2cea2a5..20d1dcf 100644 --- a/tutor/templates/k8s/volumes.yml +++ b/tutor/templates/k8s/volumes.yml @@ -1,4 +1,20 @@ -{% if ACTIVATE_ELASTICSEARCH %} +{% if RUN_CADDY %} +--- +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: caddy + labels: + app.kubernetes.io/component: volume + app.kubernetes.io/name: caddy +spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi +{% endif %} +{% if RUN_ELASTICSEARCH %} --- apiVersion: v1 kind: PersistentVolumeClaim @@ -14,7 +30,7 @@ spec: requests: storage: 2Gi {% endif %} -{% if ACTIVATE_MONGODB %} +{% if RUN_MONGODB %} --- apiVersion: v1 kind: PersistentVolumeClaim @@ -30,7 +46,7 @@ spec: requests: storage: 5Gi {% endif %} -{% if ACTIVATE_MYSQL %} +{% if RUN_MYSQL %} --- apiVersion: v1 kind: PersistentVolumeClaim @@ -46,15 +62,15 @@ spec: requests: storage: 5Gi {% endif %} -{% if ACTIVATE_RABBITMQ %} +{% if RUN_REDIS %} --- apiVersion: v1 kind: PersistentVolumeClaim metadata: - name: rabbitmq + name: redis labels: app.kubernetes.io/component: volume - app.kubernetes.io/name: rabbitmq + app.kubernetes.io/name: redis spec: accessModes: - ReadWriteOnce diff --git a/tutor/templates/kustomization.yml b/tutor/templates/kustomization.yml index 1fdb9cc..a337c7f 100644 --- a/tutor/templates/kustomization.yml +++ b/tutor/templates/kustomization.yml @@ -4,7 +4,6 @@ kind: Kustomization resources: - k8s/namespace.yml - k8s/deployments.yml -- k8s/ingress.yml - k8s/jobs.yml - k8s/services.yml - k8s/volumes.yml @@ -23,6 +22,9 @@ commonLabels: {{ patch("kustomization-commonlabels")|indent(2) }} configMapGenerator: +- name: caddy-config + files: + - apps/caddy/Caddyfile - name: openedx-settings-lms files:{% for file in "apps/openedx/settings/lms"|walk_templates %} - {{ file }}{% endfor %} @@ -35,6 +37,9 @@ configMapGenerator: - name: nginx-config files:{% for file in "apps/nginx"|walk_templates %} - {{ file }}{% endfor %} +- name: redis-config + files: + - apps/redis/redis.conf {{ patch("kustomization-configmapgenerator") }} {{ patch("kustomization") }} \ No newline at end of file diff --git a/tutor/templates/local/docker-compose.jobs.yml b/tutor/templates/local/docker-compose.jobs.yml index 48e526d..b3cc0dc 100644 --- a/tutor/templates/local/docker-compose.jobs.yml +++ b/tutor/templates/local/docker-compose.jobs.yml @@ -5,7 +5,7 @@ services: image: {{ DOCKER_IMAGE_MYSQL }} entrypoint: [] command: ["echo", "done"] - depends_on: {{ [("mysql", ACTIVATE_MYSQL)]|list_if }} + depends_on: {{ [("mysql", RUN_MYSQL)]|list_if }} lms-job: image: {{ DOCKER_IMAGE_OPENEDX }} @@ -16,7 +16,7 @@ services: - ../apps/openedx/settings/lms/:/openedx/edx-platform/lms/envs/tutor/:ro - ../apps/openedx/settings/cms/:/openedx/edx-platform/cms/envs/tutor/:ro - ../apps/openedx/config/:/openedx/config/:ro - depends_on: {{ [("mysql", ACTIVATE_MYSQL)]|list_if }} + depends_on: {{ [("mysql", RUN_MYSQL)]|list_if }} cms-job: image: {{ DOCKER_IMAGE_OPENEDX }} @@ -27,7 +27,7 @@ services: - ../apps/openedx/settings/lms/:/openedx/edx-platform/lms/envs/tutor/:ro - ../apps/openedx/settings/cms/:/openedx/edx-platform/cms/envs/tutor/:ro - ../apps/openedx/config/:/openedx/config/:ro - depends_on: {{ [("mysql", ACTIVATE_MYSQL)]|list_if }} + depends_on: {{ [("mysql", RUN_MYSQL)]|list_if }} forum-job: image: {{ DOCKER_IMAGE_FORUM }} @@ -36,6 +36,12 @@ services: MONGODB_AUTH: "{% if MONGODB_USERNAME and MONGODB_PASSWORD %}{{ MONGODB_USERNAME}}:{{ MONGODB_PASSWORD }}@{% endif %}" MONGODB_HOST: "{{ MONGODB_HOST }}" MONGODB_PORT: "{{ MONGODB_PORT }}" - depends_on: {{ [("elasticsearch", ACTIVATE_ELASTICSEARCH), ("mongodb", ACTIVATE_MONGODB)]|list_if }} + depends_on: {{ [("elasticsearch", RUN_ELASTICSEARCH), ("mongodb", RUN_MONGODB)]|list_if }} + + android-job: + image: {{ DOCKER_IMAGE_ANDROID }} + volumes: + - "../android/:/openedx/config/" + - "../data/android/:/openedx/data/" {{ patch("local-docker-compose-jobs-services")|indent(4) }} \ No newline at end of file diff --git a/tutor/templates/local/docker-compose.prod.yml b/tutor/templates/local/docker-compose.prod.yml index e3a453b..c184419 100644 --- a/tutor/templates/local/docker-compose.prod.yml +++ b/tutor/templates/local/docker-compose.prod.yml @@ -1,23 +1,36 @@ version: "3.7" services: + {% if RUN_CADDY %} + # Web proxy for SSL termination + caddy: + image: {{ DOCKER_IMAGE_CADDY }} + restart: unless-stopped + ports: + - "80:80" + {% if ENABLE_HTTPS %}- "443:443"{% endif %} + volumes: + - ../apps/caddy/Caddyfile:/etc/caddy/Caddyfile:ro + {% if ENABLE_HTTPS %}- ../../data/caddy:/data{% endif %} + {% endif %} + + # Web server nginx: image: {{ DOCKER_IMAGE_NGINX }} restart: unless-stopped + {% if not RUN_CADDY %} ports: - "{{ NGINX_HTTP_PORT }}:80" - - "{{ NGINX_HTTPS_PORT }}:443" - {% if not WEB_PROXY %} + {% endif %} + {% if RUN_CADDY and not ENABLE_HTTPS %} networks: default: + # These aliases are for internal communication between containers when running locally with *.local.overhang.io hostnames. aliases: - "{{ LMS_HOST }}" {{ patch("local-docker-compose-nginx-aliases")|indent(10) }} {% endif %} volumes: - ../apps/nginx:/etc/nginx/conf.d/:ro - - ../../data/openedx-media:/var/www/openedx-media:ro - {% if ACTIVATE_HTTPS %}- ../../data/letsencrypt:/etc/letsencrypt/:ro{% endif %} - {{ patch("local-docker-compose-nginx-volumes")|indent(6) }} - depends_on: {{ [("lms", ACTIVATE_LMS), ("cms", ACTIVATE_CMS)]|list_if }} + depends_on: {{ [("lms", RUN_LMS), ("cms", RUN_CMS)]|list_if }} {{ patch("local-docker-compose-prod-services")|indent(2) }} \ No newline at end of file diff --git a/tutor/templates/local/docker-compose.yml b/tutor/templates/local/docker-compose.yml index f9118c8..1d0de01 100644 --- a/tutor/templates/local/docker-compose.yml +++ b/tutor/templates/local/docker-compose.yml @@ -3,13 +3,7 @@ services: ############# External services - {% if ACTIVATE_MEMCACHED %} - memcached: - image: {{ DOCKER_IMAGE_MEMCACHED }} - restart: unless-stopped - {% endif %} - - {% if ACTIVATE_MONGODB %} + {% if RUN_MONGODB %} mongodb: image: {{ DOCKER_IMAGE_MONGODB }} # Use WiredTiger in all environments, just like at edx.org @@ -19,7 +13,7 @@ services: - ../../data/mongodb:/data/db {% endif %} - {% if ACTIVATE_MYSQL %} + {% if RUN_MYSQL %} mysql: image: {{ DOCKER_IMAGE_MYSQL }} command: mysqld --character-set-server=utf8 --collation-server=utf8_general_ci @@ -30,7 +24,7 @@ services: MYSQL_ROOT_PASSWORD: "{{ MYSQL_ROOT_PASSWORD }}" {% endif %} - {% if ACTIVATE_ELASTICSEARCH %} + {% if RUN_ELASTICSEARCH %} elasticsearch: image: {{ DOCKER_IMAGE_ELASTICSEARCH }} command: ["elasticsearch", "-Xms{{ ELASTICSEARCH_HEAP_SIZE }}", "-Xmx{{ ELASTICSEARCH_HEAP_SIZE }}", "--cluster.name=openedx", "--bootstrap.mlockall=true"] @@ -43,15 +37,17 @@ services: - ../../data/elasticsearch:/usr/share/elasticsearch/data {% endif %} - {% if ACTIVATE_RABBITMQ %} - rabbitmq: - image: {{ DOCKER_IMAGE_RABBITMQ }} + {% if RUN_REDIS %} + redis: + image: {{ DOCKER_IMAGE_REDIS }} volumes: - - ../../data/rabbitmq:/var/lib/rabbitmq + - ../../env/redis/redis.conf:/openedx/redis/config/redis.conf:ro + - ../../data/redis:/openedx/redis/data + command: redis-server /openedx/redis/config/redis.conf restart: unless-stopped {% endif %} - {% if ACTIVATE_SMTP %} + {% if RUN_SMTP %} smtp: image: {{ DOCKER_IMAGE_SMTP }} restart: unless-stopped @@ -59,7 +55,7 @@ services: ############# Forum - {% if ACTIVATE_FORUM %} + {% if RUN_FORUM %} forum: image: {{ DOCKER_IMAGE_FORUM }} environment: @@ -68,17 +64,17 @@ services: MONGODB_HOST: "{{ MONGODB_HOST }}" MONGODB_PORT: "{{ MONGODB_PORT }}" restart: unless-stopped - depends_on: {{ [("elasticsearch", ACTIVATE_ELASTICSEARCH), ("mongodb", ACTIVATE_MONGODB)]|list_if }} + depends_on: {{ [("elasticsearch", RUN_ELASTICSEARCH), ("mongodb", RUN_MONGODB)]|list_if }} {% endif %} ############# LMS and CMS - {% if ACTIVATE_LMS %} + {% if RUN_LMS %} lms: image: {{ DOCKER_IMAGE_OPENEDX }} environment: SERVICE_VARIANT: lms - GUNICORN_WORKERS: {{ OPENEDX_LMS_GUNICORN_WORKERS }} + UWSGI_WORKERS: {{ OPENEDX_LMS_UWSGI_WORKERS }} SETTINGS: ${EDX_PLATFORM_SETTINGS:-tutor.production} restart: unless-stopped volumes: @@ -88,22 +84,21 @@ services: - ../../data/lms:/openedx/data - ../../data/openedx-media:/openedx/media depends_on: - {% if ACTIVATE_MYSQL %}- mysql{% endif %} - {% if ACTIVATE_ELASTICSEARCH %}- elasticsearch{% endif %} - {% if ACTIVATE_FORUM %}- forum{% endif %} - {% if ACTIVATE_MEMCACHED %}- memcached{% endif %} - {% if ACTIVATE_MONGODB %}- mongodb{% endif %} - {% if ACTIVATE_RABBITMQ %}- rabbitmq{% endif %} - {% if ACTIVATE_SMTP %}- smtp{% endif %} + {% if RUN_MYSQL %}- mysql{% endif %} + {% if RUN_ELASTICSEARCH %}- elasticsearch{% endif %} + {% if RUN_FORUM %}- forum{% endif %} + {% if RUN_MONGODB %}- mongodb{% endif %} + {% if RUN_REDIS %}- redis{% endif %} + {% if RUN_SMTP %}- smtp{% endif %} {{ patch("local-docker-compose-lms-dependencies")|indent(6) }} {% endif %} - {% if ACTIVATE_CMS %} + {% if RUN_CMS %} cms: image: {{ DOCKER_IMAGE_OPENEDX }} environment: SERVICE_VARIANT: cms - GUNICORN_WORKERS: {{ OPENEDX_CMS_GUNICORN_WORKERS }} + UWSGI_WORKERS: {{ OPENEDX_CMS_UWSGI_WORKERS }} SETTINGS: ${EDX_PLATFORM_SETTINGS:-tutor.production} restart: unless-stopped volumes: @@ -113,26 +108,25 @@ services: - ../../data/cms:/openedx/data - ../../data/openedx-media:/openedx/media depends_on: - {% if ACTIVATE_MYSQL %}- mysql{% endif %} - {% if ACTIVATE_ELASTICSEARCH %}- elasticsearch{% endif %} - {% if ACTIVATE_MEMCACHED %}- memcached{% endif %} - {% if ACTIVATE_MONGODB %}- mongodb{% endif %} - {% if ACTIVATE_RABBITMQ %}- rabbitmq{% endif %} - {% if ACTIVATE_SMTP %}- smtp{% endif %} - {% if ACTIVATE_LMS %}- lms{% endif %} + {% if RUN_MYSQL %}- mysql{% endif %} + {% if RUN_ELASTICSEARCH %}- elasticsearch{% endif %} + {% if RUN_MONGODB %}- mongodb{% endif %} + {% if RUN_REDIS %}- redis{% endif %} + {% if RUN_SMTP %}- smtp{% endif %} + {% if RUN_LMS %}- lms{% endif %} {{ patch("local-docker-compose-cms-dependencies")|indent(6) }} {% endif %} ############# LMS and CMS workers - {% if ACTIVATE_LMS %} + {% if RUN_LMS %} lms-worker: image: {{ DOCKER_IMAGE_OPENEDX }} environment: SERVICE_VARIANT: lms SETTINGS: ${EDX_PLATFORM_SETTINGS:-tutor.production} C_FORCE_ROOT: "1" # run celery tasks as root #nofear - command: ./manage.py lms celery worker --loglevel=info --hostname=edx.lms.core.default.%%h --maxtasksperchild 100 --exclude-queues=edx.cms.core.default + command: celery worker --app=cms.celery --loglevel=info --hostname=edx.lms.core.default.%%h --maxtasksperchild 100 --exclude-queues=edx.cms.core.default restart: unless-stopped volumes: - ../apps/openedx/settings/lms/:/openedx/edx-platform/lms/envs/tutor/:ro @@ -144,14 +138,14 @@ services: - lms {% endif %} - {% if ACTIVATE_CMS %} + {% if RUN_CMS %} cms-worker: image: {{ DOCKER_IMAGE_OPENEDX }} environment: SERVICE_VARIANT: cms SETTINGS: ${EDX_PLATFORM_SETTINGS:-tutor.production} C_FORCE_ROOT: "1" # run celery tasks as root #nofear - command: ./manage.py cms celery worker --loglevel=info --hostname=edx.cms.core.default.%%h --maxtasksperchild 100 --exclude-queues=edx.lms.core.default + command: celery worker --app=cms.celery --loglevel=info --hostname=edx.cms.core.default.%%h --maxtasksperchild 100 --exclude-queues=edx.lms.core.default restart: unless-stopped volumes: - ../apps/openedx/settings/lms/:/openedx/edx-platform/lms/envs/tutor/:ro diff --git a/tutor/templates/local/proxy/apache2/openedx.conf b/tutor/templates/local/proxy/apache2/openedx.conf deleted file mode 100644 index ac88a4a..0000000 --- a/tutor/templates/local/proxy/apache2/openedx.conf +++ /dev/null @@ -1,39 +0,0 @@ -{% if ACTIVATE_HTTPS %} - - ServerName {{ LMS_HOST }} - Redirect / https://{{ LMS_HOST }}/ - - - ServerName preview.{{ LMS_HOST }} - Redirect / https://preview.{{ LMS_HOST }}/ - - - ServerName {{ CMS_HOST }} - Redirect / https://{{ CMS_HOST }}/ - - - - ServerName {{ LMS_HOST }} - ServerAlias preview.{{ LMS_HOST }} {{ CMS_HOST }} - SSLEngine on - SSLCertificateFile /etc/letsencrypt/live/{{ LMS_HOST }}/fullchain.pem - SSLCertificateKeyFile /etc/letsencrypt/live/{{ LMS_HOST }}/privkey.pem - - ProxyPreserveHost On - ProxyRequests On - ProxyPass / http://localhost:{{ NGINX_HTTPS_PORT }}/ - ProxyPassReverse / http://localhost:{{ NGINX_HTTPS_PORT }}/ - -{% else %} - - ServerName {{ LMS_HOST }} - ServerAlias preview.{{ LMS_HOST }} {{ CMS_HOST }} - - ProxyPreserveHost On - ProxyRequests On - ProxyPass / http://localhost:{{ NGINX_HTTP_PORT }}/ - ProxyPassReverse / http://localhost:{{ NGINX_HTTP_PORT }}/ - -{% endif %} - -{{ patch("proxy-apache") }} diff --git a/tutor/templates/local/proxy/nginx/openedx.conf b/tutor/templates/local/proxy/nginx/openedx.conf deleted file mode 100644 index 987cc66..0000000 --- a/tutor/templates/local/proxy/nginx/openedx.conf +++ /dev/null @@ -1,36 +0,0 @@ -server { - listen 80; - server_name {{ LMS_HOST }} preview.{{ LMS_HOST }} {{ CMS_HOST }}; - - server_tokens off; - location / { - proxy_set_header X-Forwarded-Proto $scheme; - proxy_set_header X-Forwarded-Port $server_port; - proxy_set_header X-Forwarded-For $remote_addr; - proxy_set_header Host $http_host; - proxy_redirect off; - proxy_pass http://localhost:{{ NGINX_HTTP_PORT }}; - } -} - -{% if ACTIVATE_HTTPS %} -server { - listen 443 ssl; - server_name {{ LMS_HOST }} preview.{{ LMS_HOST }} {{ CMS_HOST }}; - - ssl_certificate /etc/letsencrypt/live/{{ LMS_HOST }}/fullchain.pem; - ssl_certificate_key /etc/letsencrypt/live/{{ LMS_HOST }}/privkey.pem; - - server_tokens off; - location / { - proxy_set_header X-Forwarded-Proto $scheme; - proxy_set_header X-Forwarded-Port $server_port; - proxy_set_header X-Forwarded-For $remote_addr; - proxy_set_header Host $http_host; - proxy_redirect off; - proxy_pass http://localhost:{{ NGINX_HTTPS_PORT }}; - } -} -{% endif %} - -{{ patch("proxy-nginx") }} \ No newline at end of file